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CONTROLLED EQUILIBRIUM SELECTION IN 
STOCHASTICALLY PERTURBED DYNAMICS 

ARI ARAPOSTATHIS, ANUP BISWAS, AND VIVEK S. BORKAR 


Abstract. We consider a dynamical system with finitely many equilibria and perturbed by small 
noise, in addition to being controlled by an ‘expensive’ control. The controlled process is optimal 
for an ergodic criterion with a running cost that consists the sum of the control effort and a 
penalty function on the state space. We study the optimal stationary distribution of the controlled 
process as the variance of the noise becomes vanishiirgly small. It is shown that depending on 
the relative magnitudes of the noise variance and the ‘running cost’ for control, one can identify 
three regimes, in each of which the optimal control forces the invariant distribution of the process 
to concentrate near equilibria that can be characterized according to the regime. We also obtain 
moment bounds for the optimal stationary distribution. Moreover, we show that in the vicinity of 
the points of concentration the density of optimal stationary distribution approximates the density 
of a Gaussian, and we explicitly solve for its covariance matrix. 


1. Introduction 

The study of dynamical systems has a long and profound history. A lot of effort has been devoted 
to understand the behavior of the system when it is perturbed by an additive noise Pll71l32] . Small 
noise diffusions have found applications in climate modeling HE], electrical engineering [11113], 
finance |16] and many other areas. Recent work on ‘stochastic resonance’ (see, e.g., m) introduces 
an additional external input to the dynamics that may be viewed as a control. This is the main 
motivation for the study the model we introduce next. 

1.1. The model. In this paper we consider a controlled dynamical system with small noise, which 
is modelled as a d—dimensional controlled diffusion X = [Ai,... governed by the stochastic 

integral equation 

Xt = Xo+ f {m{Xs)+eUs)ds + e''Wt, t > 0. (1.1) 

Jo 

Here all processes live in a complete probability space (H,5^,P) and the data of (jl.ip satisfies the 
following: 

(a) m = [mi,..., : M'’* —)• is a bounded C°° function with bounded derivatives. 

(b) IT is a standard Brownian motion in 
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(c) U is an valued control process which is jointly measurable in {t,oj) G [0,oo) x (in 
particular it has measurable paths), and is nonanticipative: for t > s, Wt—Wg is independent 
of 

■= the completion of cr{XQ, Wr, Ur '■ r < s) relative to ((?, P). 

Such a control is called admissible, and we denote the set of admissible controls by it. As 
pointed out in HD p. 18], we may, without loss of generality, assume that an admissible U 
is adapted to the natural filtration of X. 

(d) 0 < e < 1. 

(e) u > 0. 

Let IR: X be a running cost of the form 

'Jl{x,u) := i{x), (1.2) 

where £: —)• M+ is a prescribed smooth, Lipschitz function satisfying the condition: 


lim i{x) = oo. 

|ai|^cx) 

The control objective is to minimize the long run average (or ergodic) cost 


3([/) := lim sup ^ E 

T—)-oo ^ 


Uo 


X{Xs,Us)ds 


(1.3) 


over all admissible controls. We define the optimal value /3| by 


Pt := mf^ d{U ). (1.4) 

We view o as a perturbation of the o.d.e. (for ordinary differential equation) 

x{t) = m[x(t)) , (1.5) 

perturbed by the ‘small noise’ e^Wt (‘small’ because s <C 1), and a control term eUt- Since s is 
small, the optimization criterion in (II.3h implies that the control is ‘expensive’. We assume that 
the set of non-wandering points of the flow of (II.5p consists of finitely many hyperbolic equilibria, 
and that these are contained in some bounded open set which is positively invariant under the flow 
(see Hypothesis II.ip . 

For the case when the control U = 0, Freidlin and Wentzell developed a general framework for 
the analysis of small noise perturbed dynamical systems in m that is based on the theory of large 
deviations. Under a stochastic Lyapunov condition we introduce later fHypothesis ll.ip . the cost 
is finite for U = 0, ensuring in particular that the set of controls U G ii resulting in a finite value 
for 3{U) is nonempty. It is quite evident from ergodic theory that for U = 0 the limit ()1.3I) is the 
expectation of i with respect to the invariant probability measure of dni. 

The qualitative properties of the dynamics are best understood if we consider the special case 
d = 1, and m = —^ for some smooth function F: R —>• M. Then the trajectory of (II.5p converges 
to a critical point of F. In fact, generically (i.e., for x(0) in an open dense set) it converges to a 
stable one, i.e., to a local minimum. If one views the graph of T as a ‘landscape’, the local minima 
are the bottoms of its ‘valleys’. The behavior of the stochastically perturbed (albeit uncontrolled) 
version of this model, notably the analysis of where the stationary distribution concentrates, has 
been of considerable interest to physicists (see, e.g., [Ml Chapter 8] or [T71 Chapter 6]). To find the 
actual support of the limit in the case of multiple equilibria, one often looks at the large deviation 
properties of these invariant measures m- There are several studies in literature that deal with 
the large deviation principle of invariant measures of dynamical systems. Among the most relevant 
to the present are [141137] which obtain a large deviation principle for invariant measures (more 
precisely, invariant densities) of (ll.l|l under the assumption that there is a unique equilibrium 
point. This has been extended to multiple equilibria in [8]. A large deviation principle for invariant 
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measures for a class of reaction-diffusion systems is established in m- However, none of the above 
mentioned studies have any control component in their dynamics. 

The model in (jl.ll) goes a step further and considers the full-fledged optimal control version of 
this, wherein one tries to induce a preferred equilibrium behavior through a feedback control. The 
reason the latter has to be ‘expensive’ is because this captures the physically realistic situation 
that one can ‘tweak’ the dynamics but cannot replace it by something altogether different without 
incurring considerable expense. The function i captures the relative preference among different 
points in the state space. Thus, the model in (11.11) is closely related to the model of stochastic 
resonance which has applications in neuron modelling, physics, electronics, physiology, etc. We 
refer to [191 Chapter 1] for various applications in the presence of small noise. In particular our 
model is closely related to the celebrated FitzHugh-Nagumo model [27j in the presence of noise. 
The control in o should be seen as an external input. In practice it is convenient to take U to 
be periodic in time, whereas we do not impose any periodicity constraint on U. The e factor in 
the control could be interpreted as the weak modulation in [3l]. We refer the reader to [311[35] for 
a discussion on the interplay between noise variance and the control magnitude and its relation to 
stochastic resonance. Nonlinear control theory has been useful in understanding classes of systems 
that exhibit stochastic resonance [34]. Optimization theory has also been applied with the aim of 
enhancing the stochastic resonance effect for engineered systems [mill]. 

In our controlled setting we are interested in achieving a desired value of /3^, reflecting the 
desired behavior of the corresponding stationary distribution. Although one can fix a suitable 
penalty function I beforehand, we will see in Theorem II.Ill in Section [1.31 that the value of /3^, as 
well as the concentration of the stationary distribution change with v. Therefore a desired value of 
/3^ or a desired profile of the stationary distribution might be obtained for some specific values of 
V for small e. 

We also wish to point out that, since the control and noise are scaled differently, the ergodic 
control problem described can be viewed as a multi-scale diffusion problem. 

1 . 1 . 1 . Assumptions on the vector field m. Recall that a continuous-time dynamical system on a 
topological space X is specihed by a map </>*: X —)• X, where {fit} is a one parameter continuous 
abelian group action on X called the flow. A point x G X is called non-wandering if for every open 
neighborhood U oi x and every time T > 0 there exists t > T such that fit{U) CiU 0. 

Recall also that a critical point z of a smooth vector held m is called hyperbolic if the Jacobian 
matrix Dm[z) of m at z has no eigenvalues on the imaginary axis. For a hyperbolic critical point 
z of a vector held m, we let TV’s( 2 :) and Wu{z) denote the stable and unstable manifolds of its how. 

The following hypothesis on the vector held m is in effect throughout the paper. 

Hypothesis 1.1. The vector held m is bounded and smooth and satishes: 

(1) The set of non-wandering points of the how of m is a hnite set 5 = { 21 ,..., 2 ;„} of hyperbolic 
critical points. 

(2) If y and 2 ; are critical points of m, then Ws{y) and Wu{z) intersect transversally (if they 
intersect). 

(3) There exist a smooth function V: —?> M_|_ and a bounded open neighborhood of the origin 

K, C M'’* containing S, with the following properties: 

(3a) ci|xp < V(x) < € 2(1 -|- |xp) for some positive constants ci, C 2 , and all x G K,^. 

(3b) VV is Lipschitz and satishes 

(m(x),VV(x)) < — 7 |x| (1.6) 

for some 7 > 0 , and all x ^ K,^. 

Remark 1.2. The vector held m is assumed bounded for simplicity. The reader however might 
notice that the characterization of optimality (see Theorem II.4|) is based on the regularity results 
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in [3], and the hypotheses in [3l Section 4.6.1] permit m to be unbounded as long as 

hm sup —— < oo . 

|x|—>-oo t\X) 

Provided that this condition is satisfied, the assumption that the drift is bounded can be waived 
and all the results of this paper hold unaltered, with the proofs requiring no major modification. 

1.1.2. Uncontrolled dynamics and quasi-potentials. Recall the function V defined in Hvpothesis ll.il 
Since VV is Lipschitz, AV is bounded and thus (II. 6 p implies that with 


^ofix) := — A/(x) + (m(x),V/(x)) Vxe 




we have 

C.lV{x) < 7o-7kl VeG(0,1), 

for some positive constants 7 and 70 . This Foster-Lyapunov condition implies in particular that 
the process X with U = 0 has a unique invariant probability measure t/q, and 


lim — E 
T^oo T 


r\xt\dt 

Jo 


x\r]Q{dx) <— VeG(0,1) 


7 


(1.7) 


Since i is Lipschitz, (11.711 implies that there exists a constant q independent of e such that 

J idr]^ < Cl. (1.8) 

Moreover, from [ 8 ] there exists a unique Lipschitz continuous function Z > 0, such that miujgd Z = 
0, Z{x) —)• 00 as |a;| —)• 00 and 


Z{x) = inf 

ip ■ (t>(t)^Xi, Xi£S 


1 /“CO 

- / \(j){s)+ m{^{s))\^ds + Z{xi 

J Jo 


</>( 0 ) = X, 


and if Qq denotes the density of tjq, then —s^ In Qq{x) —)■ Z{x) uniformly on compact subsets of 
as e 0. The function Z is generally referred to as the quasi-potential, and plays a key role in the 
study of r]Q. 

1.2. The optimal stationary distribution. For the model in (HI) under the optimal control 
criterion in ()1.3I) . the standard method of analysis using quasi-potentials no longer applies. The hrst 
important step is to characterize the stationary probability distributions of the controlled diffusion 
under optimal controls. It is evident that optimal controls belong to the class il dehned by 

(1.9) 


jC/Gil: E 

1 - 

(M 

"W 

1 

Jo 


< 00 for all t > 0 

We state the following result concerning the existence of solutions to dn]). 

Lemma 1.3. Under any C/ € il, the diffusion in (jl.ll) has a unique strong solution. 
Proof. See Appendix 1X1 


□ 


In studying this problem, it is of course of paramount importance to assert the existence of 
an optimal stationary distribution, and ideally also prove that it is unique. A proper framework 
for this study is to consider the class of infinitesimal ergodic occupation measures, i.e., measures 
7t e R(M'^ X M*^) which satisfy 



C^[f]{x, u) 7t(dx, du) 


= 0 


V/€C(M''), 


( 1 . 10 ) 
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where denotes the class of real-valued smooth functions with compact support. Here, the 

operator —)• C(M'^ x is dehned by 

p2u 

C‘'[f]{x,u) := —Af{x) + {m{x)+£u,Vf{x)) 

for / £ C^(R'^). We adopt the usual relaxed control framework, where an admissible control 
is realized as a 'P(R‘^)-valued measurable function (for details see [U Section 2.3]). Thus if we 
disintegrate n as 

7T(dx,du) = 77(dx) u(dtt I x), 

and denote this as tt = r/ @ u, then u is a relaxed Markov control, and rj G 'P(R'^) is an invariant 
probability measure for the corresponding controlled process, provided that the diffusion under the 
control V in (jl.ip has a unique weak solution solution for all t G [0, oo) which is a Feller process. 
Dehne 

3n := / lk(x, u) 7T(dx, du) , 7T G P(R'^ X R'^). 

Let denote the set of tt G P(R'^ x R'^) satisfying (11.101) . and consider the convex minimization 
problem 

d* := inf dn- (1.11) 

Tte'P'" 

Recall that a function /: R"^ ^ R is called inf-compact if the set {x G R"* : f{x) < C} is compact 
(or empty) for every C G R. Since (x, u) £(x) -|- ^ |up is inf-compact (and the constraint is clearly 
feasible by (11.81) 1. and n i—)• 3(7r) is lower semi-continuous, it is straightforward to show that there 
exists some G which attains the infimum (jl.lip . Also using the strict convexity of the map 
u e-)• |ttp one shows that is unique. However, it seems difficult to show that under the Markov 
control associated with the diffusion has a solution (See also Remark ll.7p . For general results 
concerning this approach see 0126]. 

In the next section we adopt the dynamic programming approach to characterize the class of op¬ 
timal stationary Markov controls and the corresponding invariant probability measures. Inevitably, 
we have to stay within a reasonably large class of controls under which the diffusion has a solution. 


1.2.1. Existence of an optimal control and the HJB equation. Recall that a (precise) stationary 
Markov control is specihed as Ut = v{Xt) for a measurable function n: R'^ —)• R"^. We identify the 
stationary Markov control with the function v. Let ilsM denote the class of stationary Markov 
controls which are locally bounded and under which dOD has a unique strong solution for all 
t G [0,oo). Note that by [211 Theorem 2.5], under a control in UsMi dl.lD has a unique solution 
up to explosion time. Moreover, it is strong Feller. Linear growth of |x| is the sufficient for the 
existence of a unique strong solution for all t G [0, oo). We let Wf denote the expectation operator 
on the canonical space of the process controlled by n G iIsM, and starting at Aq = x. We say that 
V G ilsM is stable if the controlled process under v is positive recurrent, and we let C itsM 

denote the set of stable controls in ilsMj and the set of associated invariant probability measures. 
The following theorem essentially follows from m Theorem 2.2]. 


Theorem 1.4. The HJB equation for the ergodic control problem, given by 


- — AV‘^ + min 
2 meK'* 


(m -|- £u, W^) B \ \u\^ 




( 1 . 12 ) 


has no solution if /3^ > (31, while if (3^ < j3l for any such solution W the diffusion in (ini) under 
the control v = —is transient. Moreover, the following hold: 

(a) IfV^ G C^(R'^) is any solution of (|1.12p . then |VW(x)| has at most affine growth in x; 
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(b) If j3^ = (II, then (I1.12p has a unique solution G satisfying V^(0) = 0. The 

Markov control vl := —is stable, and is optimal for the ergodic control problem under 
the performance criterion in O; 

(c) A control v G ilsM is optimal only if it agrees with vl a.e. in 

Proof. The proof is contained in Appendix O □ 

Remark 1.5. Due to the smoothness of coefficients, every weak solution in G of (I1.12p 

is automatically in for any A: G N. In the interest of notational economy, we often refer to 

any such as a solution, without specifying the function space it belongs to. 

Remark 1.6. The existence and uniqueness result for the solution of (I1.12p is well known [2l[3] and 
in fact, the results in [3] hold for a more general class of HJB equations. However, we were not able 
to find any reference that establishes the verification of optimality results in Theorem 11.41 Recent 
work as in [211123] which investigates the optimal control problem, does not exactly fit our model. 
A strict growth condition for i is imposed in Assumption (H2) of [2T], which we do not require here. 
On the other hand, in [23] where convergence of the Cauchy problem is investigated, and therefore 
optimality for the ergodic control problem is addressed, a more stringent condition is imposed (see 
Hypothesis (A3)') which for a Hamiltonian that is quadratic in the gradient like ours, amounts to 
geometric ergodicity under the uncontrolled dynamics. 

The existence of a critical value for (3^ for (jl.l2p and the behavior of the solutions above or below 
this critical value are studied in detail in (20]. However, the critical value is not necessarily the 
optimal value in (II.4p . and this is what the bulk of the proof in Appendix |A] is devoted to. For 
more recent work on the relation of the critical value of an elliptic HJB equation of the ergodic 
type and the optimal value of the control problem see [22]. 

For a stationary Markov control v we define the extended generator of (jl.ll) by 

p2i/ 

■= —Af{x) + {m{x) + ev{x),Vf{x)), x G , (1.13) 

for / G C2(M'^). 

From (I1.12P it follows that 

p.2 

—+ (m, - y I +1 = . (1.14) 

Theorem 11.41 shows that jll defined in (11.41) is attained at some G it|sM independent of 

the initial condition Xq. Let denote the invariant probability measure of the diffusion under the 
control G ifsSM- We have 

(It = [ Oi{x,vt{x))r]l{dx) (1.15) 

by Birkhoff’s ergodic theorem. Moreover, since any v G ilsM which is optimal for the ergodic 
control problem agrees with a.e. in M'^, it follows that pt is the unique probability measure in 
which satisfies (|1.15p . Given this uniqueness property, we refer to pi as the optimal invariant 
probability measure, or as the optimal stationary distribution, and we let gl denote its density. We 
also refer to vl as the optimal stationary Markov control. 

Remark 1.7. We have restricted the class ilsM to locally bounded Markov controls for two reasons. 
First, it follows from the results in [18] that under any locally bounded Markov control which 
has at most affine growth, the diffusion in dni) has a unique strong solution. Second, for a 
locally bounded Markov control, any infinitesimal invariant probability measure, i.e., a probability 
measure p satisfying f Clfdp = 0 for all / G has a positive continuous density. The positivity 
of the density is crucial in asserting the uniqueness (in an a.e. sense) of the optimal Markov control. 
However, more general statements of uniqueness are possible. As shown in m a sufficient condition 
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for the positivity of the density is that the drift in in the class for some p > d. On the 

other hand, by the results in [33], if |u| G for p > d + 2, then (11.11) has a unique weak 

solution, which is moreover a Markov process. More recent results assert strong solutions and 
continuous dependence on the initial condition for locally integrable drifts [inilMlIllI- Thus, the 
uniqueness result in Theorem ll.4l lcl can be extended to a larger class of Markov controls which 
includes singular controls. 


1.3. Main results. In this section we summarize the main results of the paper. We start with the 
following dehnition. 

Definition 1.8. Let 5s C 5 denote the set of stable equilibria of (II.5p . i.e., the set of points z G S 
for which the eigenvalues of Dm{z) have negative real parts. 

We say that a set iL C is stochastically stable (or that concentrates on iL) if it is compact, 
and lime^o = 0- It is evident that the class S) of stochastically stable sets, if nonempty, is 

closed under intersections. Hence we define the minimal stochastically stable set S by © := CiKe^K. 


The behavior of pi for small e depends crucially on the parameter u. We distinguish three 
regimes: The supercritical regime (u > 1), the subcritical regime (u < 1), and the critical regime 
[v = 1). Roughly speaking, the control ‘exceeds’ the noise level in the supercritical regime, while 
the opposite is the case in the subcritical regime. In the critical regime, which is the most interesting 
and more difficult to study, the control and noise levels are equal. The main results can be grouped 
in three categories: (1) characterization of the minimal stochastically stable set & and asymptotic 
estimates of (31 for small e in the three regimes (Theorem 11.111) . (2) concentration bounds for 
pI iTheorem 11.121) . and (3) convergence of gl, under appropriate scaling, to a Gaussian density 
(Theorem 11.131) . 


Definition 1.9. For a square matrix M G let yl"’“(M) denote the sum of its eigenvalues that 

lie in the open right half complex plane. For z G S, and with := Dm{z), where as defined 
earlier Dm{z) is the Jacobian of m at z, we let Qz and be the symmetric, nonnegative definite, 
square matrices solving the pair of equations 


mJQz + QzMz = 

{Mz-Qz)^. + ^.{Mz-Qzy = -I. 


(1.16) 


By Theorem 1 1.181 which appears in Section [1.41 there exists a unique pair {Qz, of symmetric 
positive semidefinite matrices solving (ll.lOp . It is also evident by (I1.16P that is invertible. 

Throughout the paper, the symbols 0(|x|“) and o(|x|“), for o G (0,oo), denote the sets of 
functions /: R'^ —)• R having the property 

limsup < oo, and limsup = 0, 

|x|\0 Hi |2;|\0 I^I 

respectively. Abusing the notation, 0(|x|“) and o(|x|“) occasionally denote generic members of 
these sets. Thus, for example, an inequality of the form 0(|x|) < f{x) < 0(|3:|) is well defined, and 

is equivalent to the statement that limsup|3,|^o < oo, and liminf|3.|^o > —oo. 

In order to state the main results we need the following definition. 


Definition 1.10. We define the optimal control effort S* by 

S* ^ / \vl\^dpl, s>0. (1.17) 

Also define 

Zc := Argmin {^(z) + A+(Dm( ^))}, He := min [i{z) + A+{Dm{z))] , 

z&S ^65 
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:= Argmin {i{z)'\ , 

Z^Ss 

\js 

:= min 

Z^Ss 

Hz)], 

z 

:= Argmin {^( 2 ;)} , 

zdS 

5 

:= min 
z^S 

Hz)] , 


:= Argmin 1 A"*'(llm( 2 ;))} , 

5 

:= min 
z^Z 

'yA'^ {Dm{z))] 


Theorem 1.11. The minimal stochastically stable set & is a subset of S for all u > 0. Also, the 
set &, the optimal value ftl, and the optimal control effort S* depend on v as follows: 

(i) For u > 1 {‘supercritical’ regime), & <Z Z, and 

0(^2a.) < ^ 1-2 < 0(e2-), E ^/ a = 5s, 

< PI-2 < 5 + G 0(e(2-2)A2) if 2 < 2s ■ 

(ii) For u < 1 {‘subcritical’ regime), & C Zs, and 

0(e") < G 0(e") . (1.18) 

(hi) For u = 1 {‘critical’ regime), we have & C Zc, PI < 5c + C)(e^); o.nd lim^^o PI = 5c. 

Moreover, if 2c = 5s, then the lower bound in (I1.18jl holds. 

It is not hard to show that the optimal invariant measures rjl concentrate on 5 as e \ 0 (see 
Lemma I3.1ji . In Theorem 11.111 we distinguish the three regimes corresponding to different values 
of u, and provide asymptotic bounds for PI for small e. For u > 1 one can find a control U under 
which the invariant measure of the dynamics (II.ip concentrates on a point in S. Construction of 
invariant measures with similar properties is also possible for z G 5s when u < 1. The important 
difference is that for u < 1 the optimal invariant measure cannot concentrate on 5 \ 5s (see 
Lemma 13.6p . To show this fact we construct a suitable energy function for the Morse-Smale 
dynamics (see Theorem 12.21) . The analysis in the critical regime u = 1 turns out to be more subtle 
than the other two regimes. To facilitate the study of the critical regime, we identify an important 
property which concerns a singular ergodic control problem for Linear Quadratic Gaussian (LQG) 
systems f Theorem 1 1.18p . This plays a crucial role in showing that © C Z^. 

To guide the reader, we indicate the results presented in Sections [SHU which comprise the proof 
of Theorem 11.111 

Proof of Theorem \l.lli That © C 5 follows from Lemma 13.11 Part (i) follows from Lemma 13.31 
Gorollarv l4.2l and Theorem l5.7l Part (ii) follows from Lemma[33](ii), Lemma FS.Gl and Gorollarv l4.61 
and part (hi) follows from Lemma 13.31 Remark 14.71 and Theorem 15.41 □ 


The next theorem provides concentration bounds for the optimal stationary distribution in terms 
of moments. Let dist(a:,5) denote the Euclidean distance of x G from the set 5, and Br{S) := 
{y : dist(y,5) < r}. 


Theorem 1.12. For any k gN and r > 0, there exist constants, kq = Ko{k,r,iy), and ki = kpk), 
i = 1,2, such that with r{s) := wg have 

f (dist(x,5))^ 77^(dx) < Vu>0, 

JBriS) 

[ (dist(x,5))^^77^(dx) < ki 

for all £ G (0,1). 

Moreover, if D is any open set such that 5s C D, then 

gl{D-) G 0(£2-a(2-G)^ 


Vu G (0,2] 
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provided u <1, or Zc = 3s CLud v = 1, or Z = Zs and v S (1, 2). 

Proof. This follows by Lemma l4.ll Proposition 14.51 Corollary 14.61 and Remark 14.71 


□ 


Exploiting the results in Theorem 11.121 we scale the space suitably and show that the resulting 


invariant measures are also tight. In particular, we examine the asymptotic behavior of and show 


that under an appropriate spatial scaling it ‘converges’ to a Gaussian distribution in the vicinity 


of the minimal stochastically stable set. This is the subject of the next theorem. 

Theorem 1.13. Assume u G (0,2). Let z € 5, and M an open neighborhood of z whose closure 


does not contain any other elements of S. Suppose that along some sequence En \ 0 we have 
liminfj^^o > 0. Then along this sequence it holds that 



(1.19) 


uniformly on compact sets, where ‘det’ denotes the determinant, and is given by (jl.lOp . 
Proof. This follows from Theorems 15.31 and 15.71 


□ 


We present a simple example to demonstrate the results. 


Example 1.14. Let m be a vector field in M of the form m = — VT, with F a ‘double well potential’ 
given as follows: F{x) ■= ^— ^on [—10,10], with F suitably extended so that it is globally 
Lipschitz and does not have any critical points outside the interval [—10,10]. Then VT vanishes 


at exactly three points: — 1 , 0 , 2 . Of these, 0 is a local maximum, hence an unstable equilibrium 
for the o.d.e. x{t) = m{x{t)), and both —1 and 2 are local minima, hence stable equilibria thereof. 
Let l{x) = cjxj^ on [—10,10] for a suitable c > 0, modified suitably outside [—10,10] to render it 
globally Lipschitz. Note that F{0) = 0, T(—1) = — F{2) = —|. Thus x = 2 is the unique global 
minimum of F. Since ^(0) = 0, and Dm{0) = 2, the results of Theorem 11.111 indicate that: 

• in the supercritical regime 6 = { 0 }, and « .^( 0 ) = 0 for e small; 

• in the subcritical regime 6 = { — 1 }, ^(~ 1 ) = c for e small; 

• in the critical regime, we have S = {0} if c > 2, with ~ £(0) + Dm{0) = 2 for e small, 
and S = {—1} if c < 2, with « ^(—1) = c for e small. 

Next we change the data so that F{x) ^ ^ ^ + T [“10,10]. Then VF 

vanishes at exactly five points, and S = {—2, —1, 0,1, 3}. Of these, —1 and 1 are local maxima 
of F, hence unstable equilibria for the o.d.e. x{t) = m{x{t)), while the rest are stable equilibria. 
Hence Sg = {—2,0,3} Let £{x) = 5x^ — x^ — 20x^ + 16 on [—10,10]. The critical point z = 3 is 
the unique global minimum for F, which means that it is stochastically stable for the uncontrolled 
dynamics. Calculating the values of £ at 5 we obtain: i{—2) = 24, ^(—1) = 2, ^(0) = 16, 1{1) = 0, 
and £(3) = 214. Also, we have Dm{—1) = 8 , Dm{l) = 12. By Theorem 1 1.1 11 we have: 

• in the supercritical regime 6 = {1}, and /3| ~ £(1) = 0 for e small; 

• in the critical regime, we have 6 = {—1}, and /3| ~ -^(“1) + Dm{—1) = 10 for e small; 

• in the subcritical regime & = { 0 }, /3| ~ i{0) = 16 for e small. 

Note that in this example the stochastically stable sets are distinct in the three regimes. 

Remark 1.15. Theorems 11.11111.121 suergest that u = 2 is a critical value. We present an example 
with linear drift and quadratic penalty, so that explicit calculations are possible, to show that 
indeed u = 2 is a critical value. Consider a one-dimensional model with data m(x) = x and 
i{x) = {x + 1)^. Direct substitution shows that the solution of the HJB equation (see (11.141) 1 is 



2 
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= 


1 


1 + 2^2 


+ s 


2 1 + Vl + 2 e 2 


2u-2 


The closed loop drift is 


-e^VV^{x) = -Vl + 2e2x- 


2^2 


Vl + 2 e 2 


= -Vl + 2 e 2 ( x + 


2 e 2 


( 1 . 20 ) 


V 1 + 2e2 ^ 

Thus, the optimal stationary distribution r]l is Gaussian with variance (o'^)^ and mean given 
by 




-2v 


2 e 2 


m: := 


( 1 . 21 ) 


2 Vl + 2e2 ’ * ■ l + 2e2- 

Consider the the scaled distribution with density e'^ q% (^e^x + z). Let ^(m, denote the Normal 

distribution with mean m and variance 0^2 _ Wg have 

• For u £ (0, 2), fjl converges to N(0, 1 / 2 )- 

• For u = 2, fjl converges to N(—2, 1 / 2 ). 

• For u > 2, we have ^ —>■ — 00 , and thns 7)1 does not converge as e \ 0. 

Thus (ll.lOp does not hold for u > 2. 

A simple calcnlation also shows that the optimal control effort is given by 


--2 


= 


(1 + Vl + 2e2)' (ct^)2 + + + 2c2)' ^ 


2 e 2 


(1 + Vl + 2^2) Vl + 2^2 


+ m 


= 6 ' 


_2 (1 +Vl + 2e2)^ 

4Vl + 2e2 


+ 


2 c 2 


(l + 2 e 2 ) 2 - 

Thus £ 0 (e*^ 2 i"- 2 )A 2 ^ matching the analogons estimate in Theorem II. lll (i'). 

A better understanding of this can be reached by considering the limit u —)> 00 , in which case 
the dynamics are deterministic. A simple calculation shows that 

X := argmin |£(ex) + =-— 77 - 

X 1 ^ J 1 + 2^2 

Thus for a feedback control to be optimal, the point x should be asymptotically stable for the closed 
loop system. As a result, for the LQG problem, the optimal stationary distribution is centered at 
the point x for all values of v. The criticality at z/ = 2 is generic, since in the vicinity of an 
equilibrium z, solving the minimization problem we have x ~ £^'S/i{z). 

There is a similar behavior if the drift is stable. Let m{x) = —x. We obtain 


W(x) = 

PI = 


-l + Vl + 2 e 2 


2V 


x + 


2 e 2 


1 


l + 2 e 2 

2 e 2 


+ £' 


(-1 + Vl + 2e2) Vl + 2^2 
2v-2 “1 + Vl + 2e2 


= 1 - 


+ e 


2u 


1 


1 + 2 e 2 ' " 1 + Vl + 2 e 2 ' 

The closed loop drift, variance, and mean are as in (ll.20p - (jl.21l) . Using the identity 

-1 + Vl + 2e2 1 


2V 


1 + Vl + 2V 
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the optimal control effort takes the form 

• (1 + (1 + VTT^ 

(1 +Vl + 2e2)Vl + 2e2 (l + 2e2)2' 

So 9t G 

The outline of the paper is as follows. In Section 11.41 we present an important property of LQG 
systems, which plays a crucial role in the study of the critical regime and also in the proof of 
Theorem 11.131 Section 11.51 summarizes the notation used in the paper. In Section [2] we discuss 
energy functions for gradient-like flows fTheorem 12.21) . These are heavily used in the study of the 
subcritical regime. The proofs of the main results comprise Sections[3H5l Section[3]is devoted to the 
study of the minimal stochastically stable sets, Section [His primarily devoted to the proof of Theo¬ 
rem [LT21 while Section m studies the optimal stationary distribution under an appropriate scaling, 
which leads to Theorem 11.131 Appendix [A] contains the proofs of Lemma 11.31 and Theorem 11.41 
while Appendix |B] is devoted to the proof of Lemma 11.161 and Theorem 11.181 

1.4. A property of LQG systems. As mentioned earlier, the study of the critical regime, and 
also the proof of Theorem 11.131 rely on an important property of LQG systems which we describe 
next. A matrix M S is called exponentially dichotomous if it has no eigenvalues on the 

imaginary axis. Gonsider the diffusion 

dAi = {MXt + v{Xt))dt + dWt, (1.22) 

with M G exponentially dichotomous. Let ilssM denote the class of locally bounded stationary 
Markov controls v, under which the diffusion in (11.22|) has a unique strong solution, is positive 
recurrent, and satishes 

£(■«) / \v{x)\‘^ Hvidx) < oo, (1-23) 

2 jRd 

where denotes the associated invariant probability measure. 

As Theorem 11.181 below asserts, the minimal control effort, dehned by 

£* := inf £(n), 
f'SUsSM 

which is required to render the diffusion positive recurrent by controls in ilssMj equals the trace 
of the unstable spectrum of the matrix M, which was denoted as A'^(M) in Definition 11.91 This 
result is related to classical results in deterministic linear control systems and the Riccati equation 
[25 [ 128 [ [4n]. but since we could not locate it in this form in the literature, a proof is included in 
Appendix IBI 

We start with the following lemma. 

Lemma 1.16. Provided M is exponentially dichotomous, there exists a constant Cq depending only 
on M such that 

/ \x\‘^ Hy{dx) < Cq[i+ / \v{x)\‘^ Hy{dx)] VuGilssM- 

jRd V / 

Recall that a real square matrix is called Hurwitz if its eigenvalues lie in the open left half 
complex plane. We need the following definition. 
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Definition 1.17. Let M E be fixed. Let Q{M) denote the collection of all matrices G E 
such that M — G IS Hurwitz. For G E G{M), let Sq denote the (unique) symmetric solution of the 
Lyapunov equation 

(M-G)Sg + Sg(M-G)^ = -I, (1.24) 

and define 


JciM) := ^ trace(G Sg G'^) , 
MM) ;= inf Jg{M). 


(1.25) 


Let vg { x ) = —Gx for some G E It is clear that for the diffusion in (11.221) to be positive 

recurrent under the linear control vg, it is necessary that M — G be Hurwitz. If so, then the 
invariant probability distribution of the controlled diffusion is Gaussian with covariance matrix Eg 
given by (| 1.241) . It is clear then that the control effort £(ug) defined in (|1.23p satisfies £(ug) = 
£7g(M). Therefore, provided the infimum in (jl.25p is attained, then £7*(M) is the minimal control 
effort, as defined by p.23l) . required to render p.22l) positive recurrent using a linear stationary 
Markov control. Theorem 11.181 asserts that the infimum in (jl.25p is indeed attained and that 
£7*(M) = yl"*“(M). Moreover, linear stationary Markov controls are optimal for this task within the 
class ilssM- 


Theorem 1.18. Suppose that M E is exponentially dichotomous. Then the following hold: 

(a) There exists a unique positive semidefinite symmetric solution Q of the matrix Riccati equa¬ 
tion 

M'^Q + QM = Q\ (1.26) 

satisfying 

{M + -Q)'^ = -I (1.27) 

for some symmetric positive definite matrix S. Moreover, A = M — Q attains the infimum 
in (11.251) subject to (jl.24p . and it holds that 

J*{M) = A'^{M) = ^trace(Q). 

With Hy denoting the invariant probability measure of (|1.22p under a control v E ilssM; wg 
have 

inf f ^\v{x)\‘^ fiy{dx) = A'^{M). (1-28) 

Moreover, any control u* E ilssM which attains the infimum in (11.281) satisfies u*(x) = —Qx 
for almost all x in M'^. 

Let /d The equation 

^ AV(x) + (Mx, VV(x)) - \YXpl = ^ (1.29) 

has no solution if ^ > A'^{M). If ^ = A'^{M), then V{x) = ^{x,Qx) is the unique solution 
of ()1.29p satisfying F( 0 ) = 0. If fd < A^{M) and V is a solution of (jl.29p . then the 
diffusion in (|1.22l) under the control v = —VH is transient. 


(b) 


(c) 


Remark 1.19. Optimality and uniqueness of the optimal control v{x) = —Qx in Theorem ll.lSI fbi 
holds over a larger class of Markov controls. Indeed combining the results of we can replace 

‘locally bounded’ in the definition of Ussm by |i)^| E for some p > d. Then the results of 

Theorem II. ISI lbi hold for this class of controls. 
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1.5. Notation. The following notation is used in this paper. The symbols M, and C denote the 
fields of real numbers, and complex numbers, respectively. Also, N denotes the set of natural 
numbers. The Euclidean norm on is denoted by | • |, and (•, •) denotes the inner product. For 
two real numbers a and b, a A b := min(a, b) and aW b := max(o, b). For a matrix M, M"*" denotes 
its transpose, and ||M|| denotes the operator norm relative to the Euclidean vector norm. Also I 
denotes the identity matrix. 

The composition of two functions / and g is denoted hy fog. A ball of radius r > 0 in around 
a point X is denoted by Br{x), or as Br x = 0. For a compact set K, we let dist(x,il') denote 
the Euclidean distance of x G from the set K, and Br{K) := {y £ : dist{y,K) < r}. For a 

set A C M'^, we use A, A‘^, and dA to denote the closure, the complement, and the boundary of A, 
respectively. We define A: > 0, as the set of functions whose derivatives, i = 1,... , k, are 

continuous and bounded in and denote by (M'^) the subset of (M'^) with compact support. 
The space of all probability measures on a Polish space X with the Prohorov topology is denoted by 
V{X). The density of the d-dimensional Gaussian distribution with mean 0 and covariance matrix 
S is denoted by p^. 

The term domain in refers to a nonempty, connected open subset of the Euclidean space 
W^. We introduce the following notation for spaces of real-valued functions on a domain G C 
The space U’{G), p G [l,oo), stands for the usual Banach space of (equivalence classes of) 
measurable functions / satisfying Jg|/(x)|^dx < oo, and L°°{G) is the Banach space of functions 
that are essentially bounded in G. The standard Sobolev space of functions on G whose generalized 
derivatives up to order k are in LP{G), equipped with its natural norm, is denoted by 
k > 0, p > 1. 

In general if T is a space of real-valued functions on a domain G, Tioc consists of all functions / 
such that f(p G T for every p G C)T(G), the space of smooth functions on G with compact support. 
In this manner we obtain for example the space ^i^{G). 

Also Ki, K 2 , ■ ■ ■ are generic constants whose definition differs from place to place. 

2. Gradient-like flows and energy functions 

2.1. Gradient-Like Morse—Smale dynamical systems. It is well known from the theory of 
dynamical systems that if the set of non-wandering points of a flow on a compact manifold consists of 
hyperbolic hxed points, then the associated vector field is generically gradient-like (see Dehnition l2.ll 
and Theorem 12.21 belowl. This is also the case under Hypothesis 11.11 since the ‘point at inhnity’ is 
a source for the how of m. 

Recall that the index of a hyperbolic critical point 2 : G of a smooth vector held is dehned 
as the dimension of the unstable manifold yVn{z). This agrees with the number of eigenvalues of 
Dm{z) which have positive real parts. The theorem below is well known [301138] . What we have 
added in its statement is the assertion that the energy function can be chosen in a manner that its 
Laplacian at critical points of the vector held with positive index is negative. 

We start with the following dehnition. 

Definition 2.1. We say that V G is an energy function if it is inf-compact, and has a hnite 

set S = {zi ,..., Zn} of critical points, which are all nondegenerate. A C°° vector held m on is 
called gradient-like relative to an energy function V provided that the set of non-wandering points 
of its how is S, that every point in S' is a hyperbolic critical point of m, and 

(m(x),VV(x)) <0 VxGM'^\S. 

If m satishes these properties, we also say that m is adapted to V. 

Theorem 2.2. Suppose that m is a smooth vector field in for which Hypothesis } 1.1\ holds. Let 
G be any domain of of the form {x G : V < c} for some c G M, satisfying G D IC, and let 
{02 : z G S} be any set of distinct real numbers such that if z and z' are the a- and oj-limit points 
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of some trajectory, respectively, then > a^/. Then there exists a function V G C°°{G), with the 
following properties: 

(i) (m(x), VV’(x)) < 0 for all x G G\S. 

(ii) For each z £ S, there exists a neighborhood A4 of z and a symmetric matrix Qz G 

such that V{x) = + (x — z, Qz{x — z)) + o(|x — z\‘^) for all x G A4- 

(iii) AV{z) < 0, for all z £ S \ Ss, where Ss, as defined earlier, denotes the stable equilibria of 
the flow of m. 

(iv) There exists a constant Cq > 0 such that 

Co (dist(x,5) V |VV(x)|)^ < |(m(x), VV(x)>| < (dist(x,5) A |VV(x)|)^ (2.1) 

for all X £ G. 


Proof. Since m is smooth and bounded, and m{z) = 0 for z G 5, there exists a constant Gm > 0 
such that 

|M^x —m(x)| < Cm\x\'^ VxGM'^, VzG^. (2.2) 

Let z £ S he a critical point of m of index q > 0. Translating the coordinates we may assume that 
z = 0. Since m(0) = 0, then by (12.2|) . m{x) takes the form 

m{x) = Mx + 0(|xp) 


locally around x = 0, where M = Hm(0). By hypothesis M has exactly q {d — q) eigenvalues 
in the open right half (left half) complex space. Therefore since the corresponding eigenspaces 
are invariant under M, there exists a linear coordinate transformation T such that, in the new 
coordinates x = T{x), the linear map x i—?> Mx has the matrix representation M = TMT~^ and 
M = diag(Mi, — M 2 ), where Mi and M 2 are square Hurwitz matrices of dimension d — q and q 
respectively. By the Lyapunov theorem there exist positive definite matrices Qi, i = 1,2, satisfying 

MJQi+QiMi = —Id-q, 

. . (2.3) 

MJQ2+Q2M2 = -Iq, 


where lu-q and Iq are the identity matrices of dimension d — q and q, respectively. Suppose q > 0, 
and let 0 > 1 be such that 


0 trace(T"*" diag(0, < 52 ) 7 ") > trace (t"*" diag(Qi) 0)T) , (2.4) 

and define V in some neighborhood of 0 by 

V(x) := a + (x,r'''diag((5i,-^'(32 )Tx) , (2.5) 

where a is a constant to be determined later. By (12.4h we obtain AV(0) < 0, and thus (ii) holds. 
Using (j2.2K . we have 

(m(x),VV(x)) = x'^[M^T^ diag{Qi,-eQ 2 )T + T^ diag{Qi,-eQ 2 )TM]x + 0{\xf) . 
Expanding we obtain 

T^ d[agiQi,-9Q2)TM = T^ d[agiQi,-eQ2)TT-^MT 

= T^ diagiQiMi,eQ2M2)T. 


By (j2.3p we obtain 

(m(x), VV(x)) = -(x,r’'’diag(Id_q,6(/g)rx) + 0(|x|^) . 
Therefore, since 0 > 1, we have 

- |rxp + 0(|x|^) < (m(x),VV(x)> < -0|rx|^ + O(|x|^) . 


( 2 . 6 ) 
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As shown in [38] one can select any real numbers Oj and define V on 5 by setting V(zi) = a* 
as long as the following consistency condition is met: if Zi and zj are the a- and w-limit points 
of some trajectory then ai > aj. Thus V can be defined in non-overlapping neighborhoods of the 
critical points by (|2.5I) so as to satisfy (|2.6p and parts (i)-(iii) of the theorem. Since G is positively 
invariant under the flow of m, the stable and unstable manifolds of S intersect transversally by 
Hypothesis II .11 (2). and m is transversal to the boundary of dG by Hvpothesis ll.ll (3bl. this function 
can then be extended to G by the handlebody decomposition technique introduced by Smale. For 
details see [38l Theorem B] and [301 Theorem 1]. 

It is clear by (I2.5l) - (l2.6p that (j2.1l) holds in some open neighborhood of each z € 5, and thus, S 
being a finite set, it also holds in some neighborhood of M of S. Since {m, VV) is strictly negative 
on the compact set G\Af and (m(x), VV(x)) < 0 for all x ^ S, a constant Gq can be selected so 
that (12.11) holds on G. This completes the proof. □ 

The function V in Theorem 12.21 can be extended to and constructed in a manner so that it 
agrees, outside some ball, with the Lyapunov function V in Hypothesis 11.11 This is stated in the 
following lemma. 

Lemma 2.3. Under the assumptions of Theorem \2.2\ the vector field m is adapted to an energy 
function V which satisfies V = V on the complement of some open ball which contains S. Also parts 
(i) — (iv) of Theorem \2.2\ hold. and for every bounded domain G there exists a constant Gq = Go{G) 
such that m holds for all x € G. Moreover there exists a constant Cq > 0 such that with 

y{x) := max { (dist(x,5))^ A dist(x,5), |VV(x)|^ A |VV’(x)|} , 

V_{x) := min {(dist(x,5))^ A dist(x,5), |VV(x)p A |VV(x)|} , 

we have 

{Co)-^V{x) < |(m(x),VV(x))| <CoV{x) (2.7) 

Proof. Select c G M such that Gi := {x G : V < c} contains K,. Let G2 := {x G : V < 2c}. 
By Theorem 12.21 there exists V G C°°{G 2 ) with the properties stated. Without loss of generality we 
can assume that V = 2c on dG 2 [Ml Theorem B]. Let ci := sup^r^ V. Then ci < 2c by the positive 
invariance of G2, and the property (m, VV) < 0 in G2 \ Gi. We write A d B to indicate that 
A C B. Let G := {x G : V < (ci+2c)/2}, and C2 := sup^^ V. Then Gi <s G d G2, and c < C2 < 2c 
by construction. 

Let i/:: M —)• M be a smooth non-decreasing function such that 'fi{t) = t for t < ^(ci -|- 2c), 
'ijj{t) = 2c for t > 2c, and whose derivative is strictly positive on the interval [^(ci -|- 2c), 2c]. 
Similarly, let M —)■ M be a smooth non-decreasing function such that fj^t) = 0 for t < —c and 
fi^t) = t for t > C2 — 2c. Define V := if oV + o (V — 2c). By construction V agrees with V on Gi 
and with V on G^. It can also be easily verified that supg^yci (?li, VV) < 0. Thus V G C°°(M'^) is 
an energy function, and m is adapted to V according to Definition 12.11 

Since (m(x), VV(x)) < 0 for all x ^ S, and V agrees with V on 1C, Theorem I2.2l ((i)~fiv) clearly 
hold. Also since (12.71) holds in some neighborhood of S by (I2.5I) - (I2.6I) . then, in view of the linear 
growth of (m(x), VV(x)) / 0 in (jl.6l) . and the assumptions on the growth of V in Hypothesis 11.11 
(j2.7l) also holds on □ 


3. Minimal stochastically stable sets 

Recall that /3J denotes the optimal value of (II.3p . rjl denotes the stationary distribution of the 
process X under the optimal stationary Markov control vl, and qI denotes its density. These 
definitions are fixed throughout the rest of the paper. Also recall the definition of the extended 
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generator in (11.131) . and the definition of (R in (II.2p . For a stationary Markov control u, we use the 
notation 

lR[u](x) := lR(x,i;(x)) = £{x)^\v(x)\'^ . (3.1) 

Throughout the rest of the paper V is a smooth function that satisfies (i)-(iv) in Theorem 12.21 
and agrees with V in Hypothesis 11.11 on the complement of some open ball which contains S 
fLemma 12.31) . We refer to V as the energy function. 

We start the analysis with the following lemma which asserts that r]l concentrates on 5 as e 0. 

Lemma 3.1. The family e € (0,1)} is tight, and any sub-sequential limit as e \ 0 has support 
on S. 


Proof. Recall that t/q denotes the invariant probability measure of (|l.ll) under the control U = 0. 
Define 


By (11.81) we have 


Po ■= [ 

jRd, 

[ £{x)ril{dx) < fil < PI < Cl VeG(0,1). 


(3.2) 


Since £ is inf-compact, (13.2p implies that {ql, e G (0,1)} is tight. Let ptix) denote the solution of 
()1.5p starting at x G at t = 0, i.e., (t)o{x) = x. If Cm denotes a Lipschitz constant of m and 
Xq = X, we have 


iXi- 


4>t{x)\ < Cm [ - (?!)s(x)| ds + e [ \vl{Xs)\ds + £''\Wt\. 

Jo Jo 


Hence applying Gronwall’s inequality we obtain from (j3.3p that 


sup \Xs-4>s{x)\ < / |<(X^)| ds-(-s'" sup iWsI ) . 


se[o,i] 

In turn, for any (5 > 0, (13.41) implies that 


S<t 


(3.3) 


(3.4) 


\Xt - (j)t{x)\ >dj< 
for t > 0. By Jensen’s inequality we obtain 
P, 


(5e 


— Omt 


fJ<(X,)|ds > 


-FPa;( sup IW,! > 


6e 


-Cmt 


s<t 


2e" 


C 5e 

(K{W)|d.>_ 


< Pi 


ft x2 -2Cmt 

I l<(W)l=d. > 


< 


4te^ 


-.2Cmt 




t\vt{Xs)? 

Jo 


ds 


Therefore for any compact set K C we have 


IK 


,(\Xt - (t)t{x)\ > 5) ql{dx) < 


2Cmt 


<52 


[ \vl{x)\‘^ ql{dx) 

sup Px(sup|Wsl > 

xeK \s<t 2e 




(3.5) 


It is clear that the right hand side of (13.51) tends to 0 as e \ 0. Thus for any compact set K C M'^, 
and any Lipschitz function / G it holds that 


IK 


f{Mx))\ifMx) 0 . 


(3.6) 
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On the other hand, since r]l is an invariant probability measure, we have 

[ ElHfiXt)]tidx) = f yfGChiR’^), yt>0. (3.7) 

JKd, J^d 

Let f] G be any limit of along some sequence {Sn}, with \ 0 as n —oo. By (I3.6p - (|3.7p . 

the tightness of {lyf, e G (0,1)}, and a standard triangle inequality, we obtain 

/ f{Mx))'nidx) = [ f{x)fj{dx) yt>0, (3.8) 

jRd JRd 

for all Lipschitz functions / G C6(M'^). Since the cj-limit set of any trajectory of (11.51) is contained 
in S, (|3.8p shows that fj has support on S. This completes the proof. □ 


3.1. Two Lemmas concerning the case u > 1. For z G 5, let v^, e G (0,1), denote the 
stationary Markov control defined by 


vKx) t>0, 


(3.9) 


where and are as in Definition ll.91 The controlled process, is then governed by the diffusion 

dXt = {XI,-Q,){Xt-z)dt + e’'dWt. (3.10) 


Since is Hurwitz by Theorem ll.l8l the diffusion has a stationary probability distribution 

which is Gaussian with mean z and covariance matrix S2, where S2 is as in (ll.lOj) . 

We start with the following lemma. 


Lemma 3.2. Suppose that u > 1 and z G S. Let v% he the stationary Markov control in (HH), and 
the invariant probability measure of the diffusion governed by (j3.10p . Then 



A+[Dm{z)) + , 

i{z) + 0{e^'') . 


(3.11) 


Proof. Without loss of generality assume that z = 0, and simplifying the notation we let M = 
Q = Qz, S = S2, and JP = Jl%. 

We have 


\{A'I — Q)x — m{x)\‘^ = \Qx\^+ 2(^{Qx,Mx — m{x))+ \Mx — m{x)\‘^ . (3-12) 

Since by Taylor’s theorem it holds that 

{Qx, XIx — m{x)) = (Qx, F(x)) + 0(|x|^) , 

with 

F{x) := {Fi{x),... ,Fdix)) and Fi{x) := i(x,V^mi(O)x) , 
by (12.21) and ()3.12p we obtain 

|(M — Q)x — m(x)|^ = IQx]"^ F 2(^Qx, F{x)) + 0{\x\^) . (3.13) 

As mentioned in the paragraph preceding the lemma, p,^ is Gaussian, with zero mean, and covariance 
matrix where S is the solution of (|1.27l) . Since {Qx,F{x)'^ is a homogeneous polynomial of 

degree 3 it has zero mean under the Gaussian. Also the fourth moments of are of order . It 
then follows by the estimate in (|3.13l) and Theorem II.ISI lbl that 

\ [ lii^(x)|V^(dx) = [ ^|Qx|2/l^(dx) + 0(e^^"2) 

Z jRd J^d 


A+{M) + . 


(3.14) 
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To prove the second equation in (13.111) . we use the bound 

\£(x) — £(z) — D^(z)(x — z)\ < Q |x — zp VxGM'^, z £ S . 
for some constant Ci, and since has zero mean we obtain 


(3.15) 


[ {£{x) - £{0)) fj.%dx) 


< trace(S). 


(3.16) 

By combining (I3.14|) and (I3.16|) we obtain (|3.1ip . This completes the proof. □ 

Recall the notation in Dehnition 11.101 Lemma 13.21 in conjunction with Lemma l3 .1 1 leads to: 
Lemma 3.3. It holds that 

,5* < min A~^ (Dm(z)) + 0(e^^) if u > 1, 


z^Z 


(3.17) 

(3.18) 


/3* < 5c + O(e^) if u = 1. 

Moreover, if v > 1, then 

lim 131 = Z, 
s \0 

and & <Z Z. 

Proof. Recall the function IL dehned in (|3.1I) . Since 

/3* < / 3?[u^](x)/l|(dx) Vzg5, 

the hrst inequality in (I3.17p follows by evaluating (13.lip at a point z £ Z, while the second inequality 
in p3.17p follows by evaluating (13.lip at a point z £ Z^. 

Since 

lim 131 > Z (3.19) 

£\0 

for all u > 0 by Lemma [3Tl (I3.18P follows by (I3.17P and (I3.19P when u > 1, and clearly then, in 
this case we have & <Z Z. □ 

Remark 3.4. It is worth mentioning here that if 2 : G then a control that renders {z} stochastically 
stable can be synthesized from the energy function V. Note that by Theorem I2.2l (ii). V can be 
selected so that V{z) = 0 and V{z') > 0 for all z' £ S \ {z}. Consider the control 

v^(x) := —-(m(a:) + VV(x)) , t>0. 

Then X is given by 

dXt = -VV{Xt)dt + e’'dWt, t>0. 

Let denote its unique invariant probability measure. Recall the definition in ()1.13l) . Since 

.-21/ 


it follows that 


< —||AV||oo-|VV^ 


|VVrd/l" < e"n|AV|| 


2V(x) 

Note that fp has density q’^{x) = C{e)e “7^, where C'(e) is a normalizing constant. Therefore we 
have 

[ \v’^{x)\^ fP{dx) < 2 /" (|m(x)p + IVV(x)|^)e“^/i^(dx) 
jRd jRd 

< 2 [ s~^\m{x)\‘^ fl^{dx) + s‘^''~^\\AV\\oo 
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For the last inequality we used the fact that m is bounded, m{z) = 0, and that V is locally quadratic 
around 2;. 

3.2. Results concerning stable equilibria. Recall that Ss is the collection of stable equilibrium 
points, and .^g = min^g^^ The following lemma holds for any u > 0. It shows that if 

z £ Ss then there exists a Markov stationary control with invariant measure satisfying 
/igd|u^(x)p/r^(dx) G 0(e"’) for any n G N, under which {z} is stochastically stable. 

Lemma 3.5. The following hold. 

(i) For any u > 0 and z £ Ss there exists a Markov control , and constants Eq = £0(1^) > 0, 
and Co > 0 independent of v, with the following properties. With fF denoting the invariant 
probability measure of dni) under the control , it holds that 
r 


1 1®— 


X — zf fi'^ (dx) < 


I 


v^{x)\'^ il^{dx) < 


co(l - £^) 
£2(^-1) 
Co(l - £^) 


^-coe 


^-coe 


(3.20) 


for all £ < Eq, and 


In particular, we have 


f £{x) (dx) — £{z) 
jR<i 


e\0 


0 . 


(3.21) 


sup — [ \v‘^{x)\‘^ ll‘^{dx) = 0 VuGN. 

vO JRrf 


lim 

e\0 

(ii) It holds that ,0* < 5s + o(e^) for v £ (0,2/3), and ,0* < 5s + 0(e‘^^“^) for v £ [2/3,1). 

Proof. In order to simplify the notation, we translate the origin so that z = 0, and we let M := 
Dm{ff). Let R~^ be the symmetric positive definite solution to the Lyapunov equation MR~^ + 
R~^Al'^ = —47. Thus R + RM = —AR?. Since scaling R by multiplying it with a positive 
constant smaller than 1 preserves the inequality 

M'^R + RM < -47?2 , (3.22) 

we may assume that trace(7?) < 1 and ()3.22l) holds. The sole purpose of this scaling is to simplify 
the calculations in the proof. We dehne the control by 

'£~^ (^MX — m{x)) if \Rx\ > e’'^^ , 

0 otherwise. 


'U^(x) := 


We apply the function F{x) := exp(£ {x,Rx)) to 71|e, which is defined in (I1.13|) . By 
(j3.22[) . and since trace(7?) < 1, we obtain 


{x,Rx) 


C%eF{x) = {E^''iia.cQ{R) + 2\Rx\^+ {x,{M'^R + RM)x))e~^ 


< [e^''- 2\Rx\^) 

If \Rx\ < e''!'^, then = 0, and we obtain 


{x,Rx') 


e £ 


‘Iv 


if |7?x| > 


C%eF{x) = (e^*^ trace(i7) + 2|7?2:p + 2(m(x), 7?3:)) 


{x,Rx) 

e 


< {e^'^— 2\Rx\^+ 2\Mx — m{x)\\Rx\) 


{x,Rx) 

e 


(3.23) 
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< for \Rx\ < ^\\\Rf C;;,^, (3.24) 

where in the hrst inequality we use (I3.22D . and in the second we use (12.21) . Thus selecting Eq as 

2 j 

£0 := lA(i||i?f C-') ^ 

then, provided s < Eq, (13.241) holds for all x such that \Rx\ < It follows by (I3.23P and (I3.24p 
that £|e F{x) < 0 if \Rx\ > e'^, and 

sup Fix) : \Rx\ < e'^ , E < £o} < Vc < eo • (3.25) 

Thus, by (I3.23p . (I3.24p . and (I3.25p . we obtain 

11 1 11 F^ec ) 

C%eFix) < -e^'')e 1 {|Rx|>£:-'} Vx e , (3.26) 

for all e < Eq. Note that (j3.26p is a Foster-Lyapunov equation and F is inf-compact. Therefore 
is a stable Markov control with invariant measure jF. Thus, integrating (I3.26|) with respect to the 
invariant probability measure fF, we obtain 


{x,Rx) 


J{\Rx\ > e’'} 

For any o G (0,1) we have 


e jF{dx) < e^^^ Ve<£o- 


2 |y|^ “ 

\y? < ' ' 


if |y| > a. 


(3.27) 


(3.28) 


1 -a2 

Thus using (|3.27l) . and applying (|3.28l) with a = and the inequality {x,Rx) > we 

obtain 


/ \Rx\‘^ fi^idx) < 

'\Rx\>e''/^ J\Rx\>e''/'2 I “ 




£ ' /i^(dx) 


< 


1 


1 -e^ 


f (|i2xp — £^'^) e^^/i^(dx) 

J\Rx\ > 6'' 


< e'l^ 


.2u 


^ " Ve < £o- (3.29) 


1 -e^ 

Similarly, by (I3.27p . and using the inequality (A^^ — l)|yp < N‘^{\y\‘^ — e^^) if |y| > for any 
N > 2, we obtain 


, t\t2^2u 

|72x|^/i^(dx) < e'l'^ ^ V£<£o- 


A^2 _ 1 


(3.30) 


J\Rx\> Ne'' 

Also, since by definition = 0 for |i?x| < and |h^(x)| < Cm— by (12.2p . it follows by 
(ICT1) that 


(^2 


[ \F{x)\^il%dx) < l||^2^-2g-|| 

jRd I -F 


V£<eo. (3.31) 


Then ()3.20p follows from ()3.29p and (I3.31D . by choosing a common constant cq. 
Consider the ‘scaled’ diffusion 

dXt = b^Xt)dt+ dWt, t>0, 


,, m(E''x) + E F (e" x) 

b := —^^^^ 


where 
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and let denote its invariant probability measure. It and denote the densities of and 
respectively, then = g^{x) for all x G Substituting x = in (I3.27P we deduce that 

the family of probability measures {jl^ : e G (0,1)} is tight. The (discontinuous) drift W converges to 
Mx as e \ 0, uniformly on compact sets. This implies that g^ converges, as e \ 0, to the Gaussian 
density p-^ with mean 0 and covariance matrix S, given by MS + SM"'" = —I, i.e, S = jR~^, 
uniformly on compact sets. Indeed, since is locally bounded uniformly in e G (0,1), and the family 
{p^ , c G (0,1)} is tight, the densities g^ of are locally Holder equicontinuous (see Lemma 3.2.4 
in m)- Let g be any limit point of g^ along some sequence Sn \ 0. Since {p^ : e G (0,1)} is 
tight it follows that also converges in as n ^ oo, and hence f^d g{x)dx = 1. With 

:= + (6^, V) and := + {Mx, V), and since J^d f{x) g‘^{x) dx = 0 for all / G C)?°(M'^), 

we have 



/ tf{x)) g{x)dx + 


f{x) (g(x) — g^(x)) dx (3.32) 


for all / G C)?°(M'^). It is clear that both terms on the right hand side of (|3.32l) converge to 0 as 
£ = \( 0. This implies that g is the density of the invariant probability measure of the diffusion 

dXt = MX^dt dWt, which is Gaussian as claimed. 

Since the Gaussian density has zero mean, then by uniform integrability implied by (j3.30p we 
have 

[ (Di{0)x) p^dx) -^ 0. (3.33) 

jRd £\0 

It follows by (I3.3np that for some constant Ni > 0 we have Jjgd|xp/i^(dx) < ki for all e < eo- Thus, 
using ()3.15p . we obtain 


£ [ \i{x) — i{0) — Di{0)x\ p^{dx) < KiCie'^ . 


(3.34) 


Combining (I3.33p - (l3.34p . we obtain (j3.2ip . 

Next we turn to part (ii). Consider the control n^(x) = (Mx—m(x)) for x G Then m(x) + 
ev^{x) = Mx and the associated invariant measure p^ is Gaussian with mean 0 and covariance 
matrix Using the bound in (j2.2h . we obtain 



< 



C^e ^|x|'^y^(dx) 


G 


0(e^"-2) . 


Since p^ has zero mean, using a triangle inequality, and (I3.15p . as in the proof of Lemma 13.21 we 
obtain 



(.^(x) — .^( 0 )) ^^(dx) 


< Ci trace(S). 


Since 4u — 2 < 2u for u < 1, we obtain that < .3s + 0 (e'^'^~^). On the other hand, by part (1) we 
already know that j3^ < 3s + o(e^). To complete the proof we observe that u < 4u — 2 for u > 2/3 
and u > 4u — 2 for u < 2 / 3 . □ 


3.3. Results concerning the subcritical regime. By Lemma 13.51 we can always find a stable 
admissible control such that the corresponding invariant probability measure concentrates on a 
stable equilibrium point as e 0, while keeping the ergodic cost in (II.3p bounded, uniformly in 
e G (0,1). Now we proceed to show that for u < 1, concentrates on Sg- 

Lemma 3.6. Suppose u < 1. Then pl{S \ Sg) - > 0, and lim^x q /3| = 3s- 

£:\0 

Proof. We argue by contradiction. Suppose that limsup^^Q pl{Br{z)) > 0 for some r > 0 and 
z ^ Sg. 
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In Theorem 12.21 we may select such that 7 ^ a'^ for Thus by Theorem I2.2l fiih there 

exists d > 0 be such that the interval {V{z) — Sd, V( 2 :) + Sd) contains no other critical values of V 
other than V(z). Let (/? G C^(M) be such that 

(a) ip{V{z) + y) = y for y G {V{z) - 6, V{z) + 6 ); 

(b) ip' G [0,1] on ( 12 ( 2 :) — 26, V{z) + 26 ); 

(c) if' = 0 on {V{z) - 26, V{z) + 26)^ . 

Select r > 0 such that 

sup I AV(x) — AV( 2 :)| < -|AV( 2 ;)| . (3.35) 

X&Br{z) ^ 

Note that by Theorem 12.21 and Lemma 12.31 the function V takes distinct values on S. Therefore we 
may also choose this r small enough so that 

Br{z) C {x: |V(x)-V( 2 )| <5} C .6,^(5 \ { 2 }). 

By the infinitesimal characterization of an invariant probability measure we have 

[ ^le{ipoV){x)rjt{dx) = 0 , 

jRd 

which we write as 


ip\V)AVdvl+ [ ^"{V)\VV\‘^dvt)+e [ ‘f'iV) {vt,VV) dt 

^ \jRd, Jjjd / Jjjd 

+ [ ip'{V){m,VV)drjt 

jR‘i 

Recall the definition of the optimal control effort S* in (|1.17l) . and also dehne 

C := y ^ y { V )\ VV \^ dvt ^ ' , 

e! ^ / p>'{V)AVdrf^, H := ^ [ {V)\VV\^ drjl, 

^ JW>- ^ JR'i 

and := '^1 + ‘^ 2 - Cauchy-Schwarz inequality we have 


= 0. (3.36) 


(3.37) 


[ p,'{V){vl,VV)dt 

jRd 


< llv^lloov^C" < 


(3.38) 


By Theorem [22] (iv) we have Cq (C^)^ < — /^d 7 ^^(V)(m, VV) dy^. Therefore, by (I3.36P and (13.381) 
we obtain 

Co iC)^ - e C" - e"" r < 0. (3.39) 

We write 

ei = [ ^'iV)AVdt+ [ </p'(V)AVdr?,fo (3.40) 

JBriz) JB^{z) 

Since V is inf-compact, it follows that o V is constant outside a compact set. Therefore, the 
support of (/?'(V(-)) is compact, and as a result AV is bounded on this set. By (|3.35p . (j3.40p . 
Theorem 12.2l fiiii. and since t]1[B^{S)) \ 0 as e \ 0 (by Lemma 13.ip , we obtain 


limsup (—^f) > —|AV( 2 ;) limsup ? 7 *(Rr( 2 )) > 0. (3-4:1) 

e\0 e\0 


On the other hand, since <p"{V) = 0 on some open neighborhood of S, it follows that 0 ns 
e 0. Therefore, we have limsup^^Q (—C^) > 0. However, since the discriminant of p3.39p must 
be nonnegative, we obtain 


> -2Coe^^e 


( 3 . 42 ) 
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which leads to a contradiction. Hence, r]l{S \ Ss) - > 0. This implies that liminf^^o P* > 5s; 


£\0 


□ 


which combined with Lemma l3.5l (iib results in equality for the limit as claimed. 

We revisit the subcritical regime in Corollary 14.21 to obtain a lower bound for /3|. 

It is worthwhile at this point to present the following one-dimensional example, which shows how 
the value of for small e bifurcates as we cross the critical regime. 

Example 3.7. Let d = 1, m{x) = Mx, and ^{x) = with M > 0 and L > 0. Then the 

solution to ()1.14j) is: 


= 


M + VM2 Ls^ 


2e2 


- 2 i /-2 


Note that 
respectively. 


{M + Vm 2 + Le2) . 

£(0) = 0, /3| ^ M, and —>■ oo, as e \ 0, when u > 1, u = 1, and u < 1, 


4. Concentration bounds for the optimal stationary distribution 
We start with the following lemma, which is valid for all u. 

Lemma 4.1. For any bounded domain G there exists a constant kq = such that 

(dist(x,5))^ r/^(dx) < Vu>0, VeG(0,1), 


Ip 


(4.1) 


where dist(x,5) denotes the Euclidean distance of x from the set S. 

Proof. We fix some bounded domain G which, without loss of generality contains S, and choose 
some number 6 such that 6 > sup,j,g (5 V(x). Without loss of generality assume that £{x) > Z for all 
X G otherwise we enlarge G. Let ip'. M —)• M be a smooth function such that 

(a) p{y) = y for ?/ G (-oo,5); 

(b) (p' G (0,1) on ((5,25); 

(c) (^' = 0 on [25, oo); 

(d) (p” < 0. 

Define ?i; and as in (|3.37l) by replacing p with (p, and let + Cl- ™ (I3.39P we 

obtain _ 

Co {C? - e C - e"" r < 0. (4.2) 

By Theorem I2.2l fiv) we have 

[ (dist(x,5))%^(dx) < Go^iCf. (4.3) 

J{x : V(fc) < 5} 

By an application of Young’s inequality to (j4.2|) . we obtain ^ (C)^ “ < 0 and 

hence G 0(e^^^). Thus (j4.ip follows by (|4.3I) . □ 

Corollary 4.2. Suppose u > 1. Then following hold. 

(a) The optimal control effort S* satisfies 

S* G 0(e^^^) if 5 = 5s o,n^d u > 1 , or if Zc = Zs and u = 1 , 

(4.4) 

g* G if Z <Zs and u > 1, 

and 


lim inf 

e\0 £ 


1 


2v-2 


S* > 0 if Z <Zs, and u > 1 


(4.5) 
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(b) I3l-Z> forv> 1. 

Proof. Select a domain G as in the proof of Lemma 14.11 Define , and in (l.'l-.fTh by 

replacing ip with p, and let := + ff- Then ()4.2I) holds, and thus € 0(e^^^). Recall the 

notation in Dehnition 11.101 With Ci a Lipschitz constant for i, and some hxed z ^ Z, we have 

i{x) — ^ = (£(x) — £(z)) + (£(z) — £(z)) > —Cijx — zl Vzg5, VxGM'^, 

since £(z) — £{z) > 0 for all z € S. Therefore, we obtain 

£(x)—3 > —C'£dist(x,5) VxGM'^, (4.6) 

and using the Cauchy-Schwarz inequality, and the assumption that £(x) > 5 on G‘^, we deduce 
from (|4.6I) and Theorem I2.2l fivl that 

[ £dr]l-2 > [ {£{x) - :5)dr]l > -w?- (4-7) 

JRd Jg VLo 

Thus by (14.7p and non-negativity of we have 

(4.8) 

vLo 

By (|4.7p - (l4.8p . we obtain 

< K - ! ^dr^l 

< Pl-Z+Z- [ £dr]l 

JW>- 

<Pl-Z + -%C. (4.9) 

vLo 

By an application of Young’s inequality to ()4.2I) . we obtain ^ (C)^ “ ~ < 0, and 

thus 

C' < + G'O) 

Combining (I4.9p - (I4.10I) . and using again Young’s inequality, we have 

^91 < + + (4T1) 

By Lemma [3]3] and (14.111) we obtain S* G 0(6*^^^) if 5 = 5s for u > 1, or if 5c = 5s and u = 1. 
We also obtain S* G 0(6^^*^^^"“^^) if 5 < 5s and u > 1. Thus (j4.4p holds. 

If 5 < 5s and u > 1, then Z G S \ Sg, and & C Z hy Lemma 13.31 Fix some z G S. Then 
liminfe^o Vil{Br{z)) > 0 for any r > 0. Also AV(^) < 0 by Theorem 12.21 Therefore (j3.4ip holds, 
with dim inf’ replacing the dim sup’. Expanding as in (13.401) . and arguing as in Lemma 13.61 it 
follows that (I3.41|) with dim inf’ also holds for . In fact, it easily follows that for some constant 
Ki, we have 

liminf(—,7^) > min ki(—^AV( 2 :)) . (4.12) 

z^Z 

The discriminant of the quadratic polynomial in (14.2p is nonnegative and this implies that 

e^9l > -2Coe^^e, (4.13) 


in direct analogy with (13.421) . Thus, (j4.5p follows by (j4.12l) and (j4.13p . This completes the proof of 
part (a). 
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Since C S we obtain /3* — 5 > 0(e^^^) by (14.Sp . This proves part (b), and completes 

the proof. □ 

We dehne the following scaled quantities. 

Definition 4.3. For z £ S, and as in Theorem 11.41 we define 

F/(x) := V^s’^x + z), x£R^. 

and 

■= e2y^ Vf := 

We also define the ‘scaled’ vector field and penalty by 


m|(x) := 


m{e^x + z) 


il{x) := i{e''x + z) 


The next lemma shows provides estimates for the growth of Vi4^, and VW. 

Lemma 4.4. Assume v £ (0,2], and let , and , he as in Definition \4-3\ Then 

(a) Under the restrietion that z £ Z when v £ (1,2], there exists a constant cq such that 

|VF/(x)| < co(l + |x|) Vee(0,l), VxGM'’*. (4.14) 

(b) The bound in (I4.14p also holds for W for all v £ (0,2], with no restrictions on z. 

Proof. By (|1.14p . the function Vf satishes 


-AVfix) + {mlix),VVf{x)) - - |Vy;(x)|2 = 


2 ^ ^ 2 ( 1 -^) 


{Pl-ilix)) . 


(4.15) 


Since i is Lipschitz, the gradient of the map x ^ 2 ( 1 -^) (^|(]x) — t{z)') is bounded in M'’*, uniformly 
in e G (0,1), and v £ (0,2]. Similarly, |m|(x)|, \\Dm%{x)\\ and \\D‘^m%{x)\\, are bounded in 
uniformly in e G (0,1), and u G (0,2]. By Theorem ll.lll (i). which is established in Corollary 14.21 
the constants are bounded uniformly in e G (0,1), and u G (1,2] for z £ Z. 

Applying [29l Lemma 5.1] to (I4.15jl it follows that Vf satishes (|4.14p if u G (1,2] and z £ Z. On 
the other hand, if u G (0,1], then the gradient of the right hand side of (I4.15p is bounded in 
uniformly in e G (0,1), and the restriction z £ Z is not needed. This completes the proof of 
part (a). 

Next show that (j4.14p holds for W. Fix an arbitrary z £ Z. We have 

V,W(x + z) = 


< 


£ 2 ( 1-0 


^co(l + |e ''x 


I) 




where in the inequality we use the identity Vf = (I4.14p . Since this 

proves the property for V^. This completes the proof. □ 

We continue with a version of Lemma l4.ll for unbounded domains. 

Proposition 4.5. Let v £ (0,2]. Then for any k £ N and r > 0, there exist constants and 
hi = ki{k) and K 2 = hi{k) such that with r{e) := k 2 e''^^ we have 


[ (dist(a;,5))^^r/^(dx) < 


VeG (0,1). 
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Proof. Let := Since ^^(0) = 0, by Lemma l4~4l the function is locally bounded, 

uniformly in e > 0. Applying the operator 

2y 

Cl := — A + (m-c2vl^^V) 

to the function V^^e^^ and using the identity = e^(/3^ ~ ^ ~ \ rearranging 

terms we obtain 


= 


2k 


-•^) +ke 


2y 


AV 1 - e 


2u 


V 2 
(m,VV) 


Vl/= + 2k 


vv 

~V 


+ 2k 


+ k{2k-e^^) 


|VV|^ 


(4.16) 


V ' ' V2 

By (j2.7[) , and since V has strict quadratic growth and VV is Lipschitz by Hypothesis 11.11 and V 
agrees with V outside a compact set, it follows that ^ is bounded on M'^. Therefore, in view of 
the bounds in (12.ip and ^2.71) . we can add a positive constant to V so that 

M onRL Ve>0, 


V ’ ^ V2 - V 

The constant is selected so that V > 1 on M'^. Define 

1 


(4.17) 


/~i£ _ ^ 2 — 2 A 2 uf/Q£ 

Cro •— e 


(/3|-^)- 


VV 

+ ‘^k — 


2£2A2iy 

Since £ is inf-compact, there exists tq > 0 such that Gg < 0 on We may choose ro large enough 
so that S C Brg. Let kq be a bound of PI — £ on Br^. Using this bound and p4.16l) - (l4.17p . we 
obtain 

k e^‘'AV(x)-I-(m(x), VV(x))' 


1 

t2A2u 


Cl\V'^^ e^"]{x) < v2''(x)e^'(^) 




r2l\2v 


V(x) 


for all X € M'^, and all e £ (0,1). By (12.111 we have 


e^^AV’(x)-I-(m(a:), VV(x)) < -(m(x), VV(a:)) 


(4.18) 


(4.19) 


for all X G such that dist(x,5) > with ki := \J2Cq ^||AV||cxd ■ Using (12.11) once more, if 

we define K 2 := supg^^ V) then we have 

A:(m(x), VV(x)) 


^ 2 A 2 u^. I 

e No + 


4V(x) 


<0 in |x G Br^ : dist(x,5) > 


Combining (14.181) . (14.191) . and (j4.2np . we obtain 


1 


Cl[V^^e^]{x) < 


k 


^(x)e^^*'®^(m(x),VV(x)) 


(4.20) 


(4.21) 


^2A2u *1"^ j _ ^^2A2u 

for all X G such that dist(x, 5) > r(e) := (ki V K 2 )s^^^. Let be a bound of the right hand side 
of (j4.18p on Bf.i^i;){S). This bound does not depend on e, since is locally bounded, uniformly in 
e G (0,1). Then, by p4.18p and p4.2ip we obtain 

^ e^"] (x) < K 3 + ^^4^(m(x),VV(x))v2^"^(x)e^"(*)lB-,^^(5)(x) (4.22) 


^2A2u 

for all X G and e G (0,1) 


f(e)'' 
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By the strong maximum principle, attains its infimum in in the set {x G : £{x) < /3^}. 
Therefore, V’^ is bounded below in W^, uniformly in e, by Lemma 14.41 Thus, from (j4.22jl we obtain 


L 


|(m(x),VV(x))| 


-2A2u 


V^^-\x)tidx) < 


4Nq 


A:(inf]gd e'^®) 


V e < eo . 


(4.23) 


By the strict quadratic growth of V mentioned earlier, together with (12.71) and (14.231) . there exists 
a constant K 4 , such that 


L 


1 


(dist(a;,5))^^ ^ry^(dx) < K 4 Ve G (0,1). 


This finishes the proof. 


2A2u 


□ 


Corollary 4.6. Let D be any open set such that Sg C D. The following hold. 

(a) Ifi^ = -38; then r]l{D^) G 0(e^“^) for all v G (1,2). 

(b) If V ^ ( 0 , 1 ) then 

%% G Pl-Z > 0(e"), and r,l{D^) G 0 (c 2 -a( 2 -.)) _ ( 4 , 24 ) 

Proof. Since 2 — v < 2{v A 1) for v G [1,2), then, in view of Proposition 14.51 it suffices to prove 
that r]l{Af) G for a bounded open neighborhood of 2 : G 5 \ 5s. Let (p be as in the proof 

of Lemma (3^ By Proposition 14.51 we have and /bc( 5 ) < 7 ’^(V’)AVdry^ G 

Thus 

r < iAV(z)7y,^(B,(2))+0(c2(-Ai)) (4,25) 

by (|3.35p and (|3.40p . Also 9* G 0(e^^^) by Corollary l4.2l faL which we combine with p4.25p and 
u G (1,2) to obtain 

-CoAV{z)rit{Br{z))+(D{e^) < 91 G ©(c^-) 

by (I3.42p . Thus r]^[Br{z)) G for u G (1,2). This completes the proof of part (a). 

The proof of part (b) is divided in two steps. 

Step 1. Suppose 3 = 3s- Then (j4.8p - (j4.1ip hold with 3 replaced by 3s- By Lemma [3.51 we have 
/3*—3s < . Therefore 9* G O(e^) by (14.lip , and thus G ©(c*^) by p4.10l) . Hence, /3^ — 

3s > O(e^) by (14.81) . The estimate rj^iD'^) G is obtained exactly as in Corollarv l4.6l (aL 

Step 2. Suppose 3 < 3s- By Theorem 12. 2l liiL we may construct V such that V( 2 ;) > 5 max^^ V for 
all 2 G 5 \ 5s. Let G = {x G : V(x) < 2 max^^ V} and p be as in the proof of Lemma [TTl with 
6 = 2 max 5 ^ V. We have 

3s —-^(a:) < i{z) — £{x) < Cilx — z\ V2:G5s, and x G . 


Thus 


.^(x)— 3s > max G£ |x — zl} = — dist(x,5s) VxG 

Z&Ss 


Also by Proposition 14.51 for some positive constants r and ki we obtain 

[ (^(x) - 3s)dry* > -Ki T]l{Br{z))+0{s'^’') 

zes\s. 

Therefore, splitting the integral over G and G^, we obtain as in (14.7p that 

Cf 


I 


^dry^-3s > -Ki Y vt{Br{z)) + 0{s'^’') 

zSS\Ss 


VC3 
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and since € 0 (e^), following the steps in ()4.8p - (l4.1ip we have 


and 






Co 


In view of (j4.13p and (|4.25l) we have 


zGS\Ss 


vLo 

(4.26) 

E t{Br{z))+0{s^’^). 

(4.27) 

z^S\Ss 



(4.28) 


for some positive constant K 2 - Since /?* —-Ss < by Lemma (3^ and u < 1, combining 

(I4.27P and (I4.28P we obtain S* S O(e^). Therefore by (I4.28p . we obtain r]l[Br{z)) € 

for all z G 5 \ 5s. In turn, — ^ > O(e^) by (|4.26l) . This completes the proof. □ 

Remark 4.7. If u = 1 and 5c = 5s) then following the argument in Step 2 of the proof of Corollary 14.61 
we obtain the same estimates as in (14.241) . In this case we don’t estimate S* from (I4.27p . but rather 
use Corollary I4.2l fal which asserts that S* G 0(£)- Thus ri^[Br{z)) G 0(e) by (I4.28p . which, in 
turn, implies that /3| — 5 > 0(e) by (I4.26p . 


5. Convergence of the scaled optimal stationary distributions 


We need the following definition. 


Definition 5.1. For the rest of the paper : z G 5} is some collection of nonempty, disjoint 
balls, with each "Bz centered around z, and we define Bs ■= iJzes'^z- 

Recall from Dehnition 14.31 For z G 5, we dehne the ‘scaled’ density gl(x) := e'^‘^Q%{e'^x + z), 
and denote by r)f the corresponding probability measure in We also define the ‘normalized’ 
probability density supported on 7]l{Bz) by 


qUx) 


e%(x) 

0 


if e'^x + z €Bz , 
otherwise, 


and let f/^dx) = g%{x)diX. 


Section ED which follows concerns the critical regime. The subcritical and supercritical regimes 
are treated in Section [521 


5.1. Convergence to a Ganssian in the critical regime. Recall the notation in Dehnitions 11.91 
and ll.lOl Also the scaled quantities in Definition 14.31 We start with the following lemma. 

Lemma 5.2. Assume u = 1. Fix any z G S. Then every sequence Sn\0 has a subsequence along 
which V^{x) — V^{z) converges to some Vz G C^(M'^) uniformly on compact subsets ofW^, and (51 
converges to some constant (5, and these satisfy 

^AVz{x) + {MzX,VVz{x)) - l\VVz{x)\^ = ^-i{z). (5.1) 

Moreover, for some constant cq we have 

|VK(®)| < co(l + |xl) VeG(0,l), VxGM^ (5.2) 


and 


P < yl+(M,) + ^(^). 


( 5 . 3 ) 
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Proof. If u = 1, then by (|4.15l) we obtain 

^AVf + {ml,VVf)-l\VVf\^ + il = (3t. (5.4) 

By applying [291 Lemma 5.1] to (j5.4|) and using the assumptions on the growth of m and i, it 
follows that there exists a constant cq such that 

|VF/(x)| < co(l + |x|) Vee(0,l), VxeM'^. (5.5) 

It follows by (15.4p and the bound in ()5.5p that Vf is locally bounded in C^’“(M'^), for any a G (0,1). 
It is also clear that fhl{x) —x and £|(x) — i{z), as e \ 0, uniformly over compact sets. Thus, 
taking limits in ()5.4p along some sequence \( 0 we obtain a function Vz G C^(M'^) and a constant 
/3 which satisfy dSU). The bound in (15.21) follows by (15.51) . while the bound in (15.31) follows by 
Theorem 11.181 (c) . □ 

We fix some notation. The function Vz for z G 5 denotes the limit obtained in Lemma 15.21 The 

associated ‘diffusion limit’, takes the form 

dXt = {MzXt-VVz{Xt))dt + dWt, (5.6) 

and its extended generator is denoted by 

Cz := ^A + {MzX-XVz{x),V). (5.7) 

Since (15.31) holds for all z G 5, then we must have ^ and Lemma 15.21 provides an alternate 
proof of the upper bound limsup^^Q /3| < Ho which was already shown in Lemma 13.31 In the next 
theorem we show that if liminfe^^o vVi'^z) > 0, over some sequence {Sn}, then the diffusion in 
(|5.6p is positive recurrent. 

Theorem 5.3. Assume u = 1, and let {'Bz : z G 5} be as in Definition 15. il Let Sn \i 0 he any 
sequence satisfying liminf„^oo vV {Bz) = 6z > 0 for some z G S, and {Vz,f3) G C'^{W^) x W be 
any limit point of (Vf{x) — Vf{z),/3^') along some subsequence of {Sn} (see Lemma rOI) . Recall 
Definition M.fA Then 

(a) The diffusion in dMP is positive recurrent with invariant probability measure rjz, and the 
density in Definition \5.1\ converges to the density Qz of if z, uniformly on compact subsets 
ofR'^. 

(b) The invariant probability measure ffz has finite second moments. 

(c) It holds that /3 = £{z) + A'^{Mz). 

(d) We have 

Vz{x) = \{x,Qzx), (5.8) 

and that Qz is the density of a Gaussian with mean 0 and covariance matrix Here 
{Qz,T‘z) ore the pair of matrices which solve (I1.16p . 

(e) It holds that 

liminf f (i{x) + ^\vV{x)\‘^')r]l'^{dx) > 6z {i{z) + A'^(Mz)) . 

Proof. In order to show that the diffusion in (15.61) is positive recurrent, we examine the scaled 
diffusion 

dXt = {mliXt) - VVfiXt)) dt + dWi. (5.9) 

Recall from Definition 15.II that and g^ denote the invariant probability measure of (|5.9p and its 
density, respectively. Let 

Cl := lA + {ml-VVf,X) 
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denote the extended generator of (15.91) . It follows by Lemma 14.11 and the Markov inequality that 
\ Bne{z)) < ^ for all n G N. Hence, {fjl^ : n G N} is a tight family of measures. By the 
Harnack inequality the family : n G N} is locally bounded, and locally Holder equicontinuous, 
and the same of course applies to : n G N}. Moreover, the tightness of : n G N} implies 
the uniform integrability of : n G N}. Select any subsequence, also denoted by {sn} along 
which ^1" converges locally uniformly, and denote the limit by Qz- By uniform integrability, 
also converges in as n —>■ oo, and hence Qz{x) dx = 1. Therefore f]z{(^x) := Qz{x) dx is 

a probability measure. Let / be a smooth function with compact support, and Cz be as in (j5.7p . 
Then 


^l"fix)Ql"ix)dx- Czfix)gzix)dx 


I 


< 


[ ^l"f{x){Ql"{x)-Qzix))dx 


+ 


[ {Cl^fix) - Czfix))Qz{x)dx 

Js.d. 


(5.10) 


Since —>■ Qz in the first term on the right hand side of ()5.10p converges to 0 as n ^ oo. 

Similarly, since fril"{x) —^ x and —)• VVz uniformly on compact subsets of M'^, the 

second term also converges to 0. Since ? 7 | is an invariant probability measure of (15.9p . by the 
definition of we have J^d f (x)(x) dx = 0, for all large enough n, which implies that 
f]^d Bzf{x)gz{x) dx = 0. Hence, is an infinitesimal invariant probability measure of (15.6p . and 
since the diffusion is regular, it is also an invariant probability measure. This proves part (a). 

Since the diffusion in (15.6p has an invariant probability measure, it follows that it is positive 
recurrent. By Lemma l4.II we have 

sup / |x|^ ? 7 ^(dx) < oo , 

ee(0,l) J{e'^x+ze'Sz} 

which implies by Fatou’s lemma that f 72 (dx) < oo. Also by Theorem 11.41 and Theo- 

rem ll.lSl fcl we must have ^ — i{z) = A'^{Mz). This completes the proof of parts (b) and (c). 

By part (c) and Theorem ll.lSI lcl the solution of (|5.ip is unique and is given by (15.8p . That gz is 
Gaussian with covariance matrix follows by the second equation in (ll.lbp . This proves part (d). 

Since Vz has at most quadratic growth by (j5.5l) . we have J^d\Vz{x)\fiz{dx) < oo. Therefore, 
with Ea, denoting the expectation operator for the process governed by (15.6p . it is the case that 
Ea;[H^(A"t)] converges as t —)• oo [211 Theorem 4.12]. Integrating both sides of (15.ip with respect to 
fjz, we deduce that 

f ^\VVz{x)\'^ fjzidx) = (3-t{z). (5.11) 

Using Fatou’s lemma, we obtain by part (d) that 

liminf / IR[u^"](x) (dx) = liminf / (i\^{x) + ^\W^'^{x)\^\fi1'^{dx) 

^n\0 £n\0 g 33^} 

> lim liminf / (".^^'‘(x) + i|VlT^"(x)|^') r/^"(‘B^) 57^"(dx) 

R-^oo en\0 J{\x\<R} ^ ' 

> Qz{A-^iMz)Ai{z)), 

where in the second inequality we use ()5.1ip . along with the hypothesis that ril^{'Bz) —>■ > 0. 
This proves part (e) and thus completes the proof. □ 


Part of the statement in Theorem 1 1.11 1 (iii) follows from the following result. 
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Theorem 5.4. Recall the definition of Zc from Theorem \l.ll[ We assume u = 1. Then, it holds 
that lim^^o fit = 3c- Also fi in (15.11) equals 3c- Moreover, for any r > 0 we have 

lim r]l{BfrZc)) = 0, and lim [ \vl{x)\‘^ ijfrdx) = 0. (5.12) 

e\o Jb^{z^) 

Proof. Since the collection {(B^} used in Theorem 15.31 was arbitrary, without loss of generality, we 
may let = Br{z). Let \ 0 be any sequence such that rjfr (^Br{z)) —)• 6z for all z G S, and 
define So '■= {z G S : 0z > 0}. Thus X^zeSo ~ Theorem 15. 3l fei we have 

lim inf > T] / ifrfr + \\frfr fr)?) frfr 

- Oz (i{z) + A'^{Dm{z))'j > 3c- (5.13) 

Z^So 

Since lim sup^^Q fil < 3c by Lemma 13.31 (I5.13P implies that lime^o fit = 3c - By Lemma 15.21 we 
have liminfg^o fil ^ fi: and ,5 < 3c by (15.31) . Therefore fi = Zc- 

Given any sequence \ 0, we can extract a subsequence also denoted by {e^} along which 
lim^^oo hfr{Br{z)) —>■ Gz for all z G S. Then (|5.13p holds. Also, by Proposition 14.51 we have 
/bc( 2 ) frx) rjfrdx) —)• 0 as e \ 0. It is then clear that both assertions in (15.121) follow by (I5.13p . □ 

It is interesting to note that (15.81) holds for any z G Zc even if lime^o = 0. This is part 

of the corollary that follows. 

Corollary 5.5. Suppose u = 1. Then for any z G Zc, we have 

yfifr)-Vf{z) ^ ]^{x,Qzx), 

uniformly on compact sets. Also, unless z G Zc, then the family (r)^ ; e € (0,1)} is not tight. 

Proof. Since fi in (15.11) equals 3c by Theorem 15.41 then, provided z G Zc, the right hand side of 
(j5.ip equals A~^{Mz). The first assertion then follows by Theorem II. ISI lcL 

If the family {fj^ : e G (0,1)} is tight, then it follows from the proof of Theorem 15.31 that the 
diffusion limit in (15.6p is positive recurrent. However, if z ^ Zc, then fi — l{z) = 3c —^(- 2 ) < A~^{Mz), 
and by the results of Theorem 11.41 and Theorem II. ISl fci. the diffusion in (|5.6p has to be transient. 
Therefore, { 17 ^} cannot be tight. □ 

Remark 5.6. It is worth examining the diffusion in (|5.6I) in the context of Example 11.141 Consider 
the example with the first set of data, and let c = 5. Then 6 = {0} and 3c = 2. Thus, for z = 0, 
we have Vz = Vq = 2x^, and the drift in (15.6p equals —2Xt. For z = —1, we have ^(—1) = 5, 
Dm{—1) = —3, and direct substitution shows that V-i = —3a;^ solves (15.11) . The associated 
diffusion in (15.6p has drift 3At, and thus it is transient. 

5.2. Convergence to a Gaussian in the subcritical and supercritical regimes. We return 
to the analysis of the subcritical and supercritical regimes in order to determine the asymptotic 
behavior of the density of the optimal stationary distribution in the vicinity of the stochastically 
stable set. In these regimes there are two scales. If we center the coordinates around a point is 6 , 
then we have V^(x) G 0(e“^|xp), and —log 0 ^(a;) G 0[e~‘^'^\x\‘^). To avoid this incompatibility we 
use the function Vz{x) = x) in the analysis, which scales correctly in space for all u. 

We have the following result. 

Theorem 5.7. Assume v G (0, 2) and let : z G 5} be as in DeRnition \5.1[ The following hold. 
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(a) Suppose that for some z G S and a sequence \ 0 holds that liminf^^oo > 0. 

Then the density in Definition \5.1\ converges as n —>■ oo (uniformly on compact sets) to 
the density of a Gaussian with mean 0 and covariance matrix given in (|1.16l) . 

(b) If v G (1, 2) and z G S \ Z, then linig^o = 0. 


Proof. The proof closely follows those of Lemma 15.21 and Theorem 15.31 Only the scaling differs. 
We summarize the essential steps. 

First, suppose u < 1. Since hminf„^oo 'n*"'i'^z) > 0 then necessarily z G 5s by Lemma (3^ We 
scale the space as i/e*", and use (|4.15l) which we write again here: 

^ |Vy/(x)p = _ ^£(3;)) , (5,14) 

By Lemma lOl VV/ = is locally bounded and has at most linear growth. We write 

(I5.14P as a HJB equation 


+ u, VVf{x)') + ^ |a|' 


= c2(l-0 




(5.15) 


The associated scaled controlled diffusion is 


dw = {mliXt) - Ut) dt + dWt. 

Taking limits in ()5.15p along some subsequence Sn \ 0, we obtain a function Vz G 
most quadratic growth satisfying 


iA 14 (a;) + min (M^ x + n, V14(a^)) + 


= 0 . 


(5.16) 
l'^) of at 

(5.17) 


The associated diffusion limit is 


dXt = {MzXt-VVziXt))dt + dWt. (5.18) 

As in Section EH ? 7 | denotes the invariant probability measure of (15.161) under the control 
Ut = —XVf{Xt), and gl its density. Following the proof of Theorem 15.31 and using Lemma EH 
we deduce that the density in Definition 15.11 converges as \ 0 to the density gz of the 
invariant probability measure of (|5.18l) . However since Mz is Hurwitz, then X^{Mz) = 0, and by 
Theorem 11.181 we obtain I 4 = 0. So in this case (I5.17P is trivial, and the covariance matrix of 
the Gaussian is the solution of (jl.l6p with Qz = 0. 

Next we assume u G (1,2), and we use the same scaling and definitions as for the subcritical 
regime, except that z G Z. It is clear that 

(ll{x) - l{z)) < ^ q. 


where denotes a Lipschitz constant of I. By Corollary 14.21 the constants ^^ (/?* — I{z)) are 
bounded, uniformly in e G (0,1). Therefore, as argued in the proof of Theorem 15.31 for every 
sequence \ 0, there exists a subsequence, also denoted as {Sn} along which 
converges to a constant /3, and H/(-) — Vf(z) converges to some Vz G C^(M'^), uniformly on compact 
sets. Taking limits in (15.151) along this subsequence, we obtain 


-AVz{x) + min 

2 'ueR‘* 


{Xlz X + u,Wz{x)) + 


/ 3 . 


(5.19) 


Recall the notation Z and Z in Definition 11.101 By Lemma 13.31 we have 

$ < Z = vain {Dm{z)) . (5.20) 

Following exactly the same steps as in the proof of Theorem 15.31 we deduce that the diffusion 
in (|5.18l) is positive recurrent, with an invariant probability measure that has finite second 
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moments, and that the density q% in Definition 15.11 converges as 6^ \ 0 to the density Qz of 
Therefore, 

A+{Dm{z)) = /3 (5.21) 

by Theorem I1.18l fci. Thus /3 = 5 = (^Dm{z)) by (I5.2U|) - (I5.21I) . This shows that unless z G Z, 

the hypothesis liminf^^oo > 0 cannot hold, thus establishing part (b) of the theorem. 

With z G Z, and /3 = 5, equation (I5.19P has a unique solution by Theorem ll.lSI fch and we 
obtain Vz(x) = ^ (x^Qz), and that Qz is the density of a Gaussian with mean 0 and covariance 
matrix with {Qz,'^z) the pair of matrices which solve (I1.16h . This completes the proof. □ 

6. Concluding remarks 

In general, Morse-Smale flows may contain hyperbolic closed orbits, and it would be desirable 
to extend the results of the paper accordingly. An energy function V as in Theorem 12.21 may be 
constructed to account for critical elements that are closed orbits [301138] . Note that under the 
control used in Remark 13.41 the optimal stationary distribution concentrates on the minimum of V. 
In the case that z G belongs to a stable periodic orbit with period Tq, we can construct V so 
that it attains its minimum on this closed orbit. In this manner, if (f>t denotes the flow of the vector 
field m, then it follows by (13.8p that under the control used in Remark 13.41 we obtain 

[ i{x)n^{dx) ^ dt. 

The same can be done in the subcritical regime, by modifying the proof of Lemma 13.51 and using 
instead the approach in Remark 13.41 We leave it up to the reader to verify that Lemma 13.11 still 
holds if the set of critical elements S contains hyperbolic closed orbits. Let us define 

iiz) ^ dt, 

4o Jo 

when z belongs to a closed orbit, and £{z) = £(z), when m(z) = 0. Then, provided Argmin^g^ £(z) 
contains only stable critical elements, then the support of the limit of the optimal stationary 
distribution lies in Sg, and this is true in any of the three regimes. However, the full analysis when 
unstable closed orbits are involved seems to be more difficult. 

Appendix A. Proofs of the results in Section 11.21 
We start with the proof of Lemma 11.31 

Proof of Lemma \ 1.31 The proof is standard. Let U be given and define Mt := E[Jp ds], 
t G M+. For T > 0, let denote the space of {5^i}-adapted processes Y defined on [0, T], having 
continuous sample paths, and satisfying E[supo<t<'r |L)P] < oo. The space (more precisely the 
set of equivalence classes in 'H'^) is a Banach space under the norm 


It is standard to show, for example following the proof of [H Theorem 2.2.2] that any solution X 
of ()l.ll) satisfies 

\\X-Xofy^2 < Kot(l + t)(l + Mt + E[|Xop])e'‘i* Vt>0, 

for some constants kq and ki that depend only on m. The existence of a pathwise unique solution 
then follows by applying the contraction mapping theorem as in [H Theorem 2.2.4]. □ 


|T| 




:= E 


sup \Yf\ 

.0<t<T 
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The rest of this section is devoted to the proof of Theorem 11.41 Without loss of generality we 
fix e = 1, and denote the optimal value (51 in (11.41) as /3*. Also throughout the rest of this section, 
without loss of generality we assume that £ > 0. 

We proceed by establishing two key lemmas, followed by the proof of Theorem 11.41 Recall the 
dehnition of Ik in (II.2p . For x G and a > 0, we define the subset il" of admissible controls by 


il“ := <U Gii: E 


u 


e-‘^^Jl{Xs,Us)ds 


< oo 


(A.l) 


where denotes the expectation under the law of {X, U), with Xq = x for the solution of 

Xt = X + f m(As)ds+ f Usds + Wt, t>0. (A.2) 

Jo Jo 

Lemma A.l. The equation 

iA14 + (m,VF„)-i|VF„|2+.g = (A.3) 

has a solution in for all a G (0,1). Moreover, for all a G (0,1), we have the following: 

(i) For some constant cq > 0, not depending on a, it holds that 

|V14(x)| < co-\/l + \x\ , and \aVa{x)\ < i{x) + ^ VxGM'^. (A.4) 

(ii) The function Va satisfies 


VJx) < inf E 


u 


uo 


e""" 17s) ds 


Vx G 


(A.5) 


(iii) With Cl the constant in we have 


inf aVoi 

{x : £{x) < q } 


inf aVa < Cl. 

mA 


Proof. In [21 Theorem 4.18, p. 177] it is proved that (IA.3I) has a solution in and it also 

shown in the proof of this theorem that there exists a constant kq > 0 which does not depend on 
a such that 

aVa{x) > —kq VxGM'^. (A.6) 

By m Theorem B.l] there exists a constant C not depending on 7? > 0 such that 


sup |V17o,| < (7(1+ sup {aVa)~ + sup + sup 

Br ^ Br+x ^^h+1 Brj^x 



from which gradient estimate in (IA.4I) follows. The structural assumption on the Hamiltonian 
h{x,p) in |21l Theorem B.l] is p i—>• h{x,p) is strictly convex for all x G and there exists some 
constant ko > 0 such that 

ko\p\^ < h{x,p) < kQ^\p\^, \Vxh{x,p)\ < {l + \p\'^) , (x,p)gM^'^. (A.7) 

This Hamiltonian corresponds to h{x,p) = — {m,p) for the equation in ()A.3p . and the hrst 

bound in (|A.7p is not satisfied. However, replacing this bound with 


ko{\p?-ki) < h{x,p) < feg ^ (IpP + fei) , 


for some constant ki > 0, the proof of [211 Theorem B.l] goes through unmodified. 

Recall the definition of if in (jl.911 . Writing ()A.3h in HJB form, and applying Ito’s formula we 
obtain 


H„(x)-e-“'E^[H„(W)] < E 


u 


e-°^X{Xs,Us)ds 


Vt>0, VC/Gil. 


(A.8) 
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Since m is bounded, then it is standard to show using (IA.2P that 


E: 


u 


sup |X(s) — xl 
0<s<t 


< llmlloo t 


\UJds 


< oo Vt/Gil, Vt>0. (A.9) 


Also, if E° denotes the expectation E^ with U = 0, then by ()A.2p we have the estimate 

E°[|At|^] < K2{1 ++ \x\‘^) < oo Vt>0, (A.10) 

for some constant K 2 - As shown in the proof of [21 Theorem 4.18, p. 177], a aVa{0) is bounded 
on (0,1), which together with the gradient estimate in (1A.4P we have already proved, provides us 
with a liberal bound of Va of the form | 14 i,( 3 :)| < C'(l + |xp) for some constant C. This combined 
with (lA.lOl) implies that e“"* E° [1 /q,(X 4)] —)■ 0 as t —)■ oo. Therefore, using (lA.Op . and the Lipschitz 
constant Ci of i, we obtain by (IA.8I) that 


aVa{x) < E' 


ae-‘^U(Xs)ds 


pOO 

< i{x) + Cl / ae“"^(||m||oo s + 2v^) ds VxGM'^, 

Jo 

which results in the estimate given in (|A.4h . where without loss of generality we use a common 
constant cq. This completes the proof of part (i). 

Let g{x,t) := \x\ + ||m||ooi + 2y/t. Multiplying both sides of (IA.9p by e“"*, strengthening the 
inequality, and applying the Holder inequality, we obtain 

e""*E^[|At|] < 5(x,t)e""*+ e"t*E^ [ e"t*|t/,|ds 

Jo 




0 

E^f 


e-“" |t/,pds 


1/2 


t—^oo 


-> 0 


(A.ll) 


for all U G il“, as defined in (lA.lD . Taking limits as t —)• oo in (IA.8I) . and using (lA.llh . and the 
bound of Va in (IA.4P together with \l{x)\ < Ci\x\ + 1-^(0)], we obtain (lA.Sp . 

We now turn to part (iii). Let 

1/2 


X{x) ■■= 


1 




mm 


Vs \y&Bi{x) - 


£{y) — (d + 1 + 2\/d II m\\ 


and 


V'(x) := 14(x) + ^ - x(3:o)(l - lx - xop) , x G Hi(xo) 
where kq > 0 is the constant in (jA.Bp . With (/)(x) := |x — xop, we have 

(m-VH„,VV^)+ = (-i AW. - (m, VW.) + i |VH„|2 - 


+ 5 |VW - x(xo)V(/>|^ - 2x^(xo)</> + 2 ko 

- x{xo) A(/> + {m,V(j)) + a{l - (j))^ 

> I - 2x^(xo) + 2/to - (d + 2\fd ||m||oo + l)x(a^o) 

> - 3x^(xo) - (d + 1 + 2Vd ||m||oo)^ in ili(xo), Va G (0,1), 

where we use (|A.3h and the fact that kq > 0. Since > 0 on dBjxJ) by ()A.6h . an application of 
the strong maximum principle shows that ^ > 0 in i?i(xo), which implies that 

aVaix) > ax{x) + kq V x G . 
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Since I is inf-compact, and therefore the same is true for x by its definition, this shows that aVa is 
inf-compact. In particular, it attains its infimum in With r/o denoting the invariant probability 
measure of the diffusion in (IA.2I) under the control U = 0, using ()A.5|) . we obtain 


inf 14 , 



< 



e-^^i{Xs) ds 


r]o{dx) 


< 


2i 

a 


(A.12) 


where the last inequality follows by (|1.8I) . One more application of the maximum principle implies 
that if Va attains its infimum at x G then £(x) < aVa(x). This together with ()A.12p implies 
part (iii). □ 


Remark A.2. We should mention, even though we don’t need it for the proof of the main theorem, 
that (lA.Sp holds with equality, and thus I 4 is indeed the value of the infinite horizon discounted 
control problem. The proof of this assertion goes as follows. Since VI 4 has at most linear growth, 
the diffusion in ()A.2p under the Markov control = —VI 4 has a unique strong solution. It is also 
clear by (jA.4p that for any a > 0 we can select a constant K,i{a) such that |VI4(a:)| < ^ 1 ( 0 ;) -|- ^x. 
Thus using a standard estimate [U Theorem 2.2.2] we obtain 




sup |X(s)|2 

.0<s<i 


< «;2(a)(l-k t^)(l + |a:p)e2* 


(A.13) 


for some constant ^ 2 ( 0 ) > 0. With T/j denoting the first exit time from applying Dynkin’s 
formula we obtain 


Va{x) = E!; 








We write 

E!^“ [e-“hA^«) )] = Ai (t, i?) + A 2 (t, i?). 

with 


Ai(t, R) := E]]“ [e-“' R) := E]]“ [e-“"« <t}] • 

Since I 4 has at most linear growth in x by ()A.4I) . it follows by (|A.13I) that 

lim limsup |Ai(t,i?)| = 0. 

t^QO 


We also have limsup^^oo |A 2 (t, i2)| = 0 by dominated convergence, since P!^“ (t/j < t) —)• 0 as 
i? —>■ 00 . Thus, taking limits first as i? —>■ 00 , and then as t —)• 00 in (IA.13h . we obtain 


14 (x) > E^ 


Jo 


e-°^X{Xs,v^{Xs))ds 


Thus the converse inequality to (IA.5h also holds. 


Recall the definition of /3| in ()1.4p , which for e = I we denote simply as /3*. Also define the class 
of controls it^ by 

Ucc := {ueii: limsup 7 ^ E^ / 3l{Xs,Us)ds 
I T-^00 T Uq 

Lemma A. 3. There exists V G which is bounded below in and satisfies 

A[V]{x) := \ XV + {m,XV) - \\XV\'^ + I = fi . (A.14) 

for 13 = (3^,. Moreover, under the Markov control Ut = —v^{Xt), with u* = —W, the diffusion in 
()A.2p is positive recurrent, and /3* = dr/*, where r/* is the invariant probability measure 

corresponding to the control u*. 
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Proof. The existence of this solution is established as a limit olVa{-) — 14i:(0), with the solution 
of (jA.3h in Lemma [A. II along some sequence cin \ 0 [21 p. 175]. It also follows from the proof from 
this convergence result that /3 < limsup^^g Q:V"q,(x) for all x € 

We first show that /3 < /3*. For this, we employ the following assertion which is a special case of 
the Hardy-Littlewood theorem [39]: For any sequence {an} of non-negative real numbers, it holds 
that 

OO 1 ^ 

limsup (1 — 0) < limsup On ■ (A.15) 

Concerning this assertion, note that if the right hand side of the above display is finite then the 
set {^} is bounded. Therefore in finite for every 6 < 1. Hence we can apply [39] 

Theorem 2.2] to obtain (IA.15p . 

Fix X G and U £ ilx- Define 




:= E 


u 



Jl{Xs,Us)ds , 


n > 1. 


and let 0 = e “. Applying (IA.15p . and with N running over the set of natural numbers, we obtain 


limsup — E! 

N^oo A 


U 


- f-N 

/ '3l{Xs,Us)ds 

Jo 


> 

lim sup 


e/'i 

> 

lim sup 


a\0 

> 

lim sup 


Q:\0 

> 

/3. 


n=l 


CXD ■ pfi 

> limsup (1-e"“)VE^ / e""" t/,)) ds 

»\0 Un-l ^ 




e""" t/,) ds 


(A.16) 


where we use the property that limsupQ,^Q aVa(x) > /3. Since U G is arbitrary, (IA.16P together 
with the definition of /3* imply that /? < /3*. Note also that (|A.16h implies that il“ C ila, for all 
cc G (0,1). 

Next, we prove the converse inequality. By (IA.4p we have jVH(x)j < cq (l + jx]). Therefore, 
since the Markov control u* := —XV{x) has at most linear growth, there exists a unique strong 
solution to ()A.4p under the control u*. Applying Ito’s formula to (|A.14p . and using (|3.ip . we obtain 


E^*[H(Atat«)] -A(x)+E, 






lk[u*](As) ds 


= /3 lEi [T A Xr] , 


where denotes the exit time from the ball of radius R > 0 around 0. Since V is bounded from 
below and T/j —)• oo a.s., as R ^ oo, using Fatou’s lemma for the integral on the left hand side, and 
then dividing by T and taking limits as T —)• oo, results in 

cT 


limsup ^ El* 

T^oo ^ 


Jo 


ds 


< /3. 


Thus (5 = Since ^ is inf-compact this also implies that the diffusion under the control u* is 
positive recurrent, and by the ergodic theorem we obtain /3* = lk[u*](x) dr/*. □ 


Proof of Theorem \1.4\ Part of the proof relies on results in [20| . These results require a Foster- 
Lyapunov type hypothesis which we verify next. Note that the operator F in m has a negative sign 
in the Laplacian so that A[ip\ = —F[ip\, where A is the operator defined in (IA.14p . So, given that 
I is inf-compact, (/Jq = 0 is an obvious choice to satisfy (A4) in [20]. Then of course —A[^po\ —oo 

as jxj —>• oo. Note that Theorem 2.2 in |2nj then asserts that V is bounded below in 
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Next, consider = —ai\/V with oi := vuifcc where K, is as in Hypothesis [LT](3) and 

V is as in Lemma 12.31 Since V agrees with V outside some compact set by Lemma 12.31 it follows 
by Hypothesis Il.ll f3i that oi > 0. Then we obtain 

iA^i + (m,V<^i)-i|V<^i|2 = ^AV-^((m,VV) + ^^i^|VV|2) 

> ^(AV-(m,VV)) on /CT 

Thus, since AV is bounded by Hypothesis ll.ll (3bL we obtain —A[ipi] —)• —oo as |x| —)• oo. It is 
also clear that 4)o{x) — (/)i(x) —)• oo as \x\ —>• oo. Thus, Hypothesis (A.4)' in [2D] is also satisfied. 
Therefore, as shown in 1201 Theorem 2.1], there exists some critical value A* such that (I1.12|) has 
no solution for /3 > A*. Also by Theorem 2.2 and Corollary 2.3 in |2D|, the following hold: if H is a 
solution for (5 < \*, then under the control v = — VH, the diffusion is transient. For (3 = X* there 
exists a unique solution H = lA (up to an additive) constant, and under the control u* = —VIA 
the diffusion 

Xt = Xo+ [ {m{Xs) + v{Xs))ds + Wt, t > 0, 

Jo 

is positive recurrent. It is clear then that Lemma lA.31 implies that A* = /3*. 

Part (a) of the theorem, was established in the proof of Lemma IA.31 it also follows directly 
by |29[ Lemma 5.1]. 

The uniqueness of the solution for /3 = (3^ follows by the results in [23] discussed above, while 
the rest of the assertions in part (b) follow by Lemma I A. 31 

We now turn to part (c). Suppose v G ilssM is an optimal stationary Markov control, and let i] 
denote the associated invariant probability measure. In particular, 

[ {^{x) + \\v{x)\'^) i^x = /3*. (A.17) 

It is straightforward to check from (IA.3p that 

iAI4 + (m + f;,VI4) + i]f}]2+.g = al4+ijVlA + f)|2. (A.18) 

By the very definition of ilssMj the diffusion in (IA.2P under the control D is a strong Markov process. 
By applying Dynkin’s formula to (lA.lSp we obtain 


K[Va{XTA^^)]-Va{x)+K 


fTArji 

/ {i + \\v\^){Xs)ds 
Jo 

fTArji . . 

(alA + ilVlA + f;]2j(X,)ds 


= 


. (A.19) 


The hypothesis that v is optimal, and thus has finite control effort, implies by (IA.9P that 


E 


V 

X 


sup lA(s)l 
.0<s<t 


< OO 


VxGM'^, Vt>0. 


(A.20) 


Therefore, since jlA(x)| < co(l + Jx]) for some constant cq = co(q;) by (jA.4ll . then by dominated 
convergence implied by (IA.20p we obtain E!^ [lAf(X 7 ’ATfl)] —^ as 7? —)• oo. Thus taking 

limits in (IA.19I) as 7? —)• oo, using monotone convergence for the integrals, it follows that ()A.19p 
holds with T A Xfi replaced by T. 

By (|A.17h we obtain lim'r^oo y IE"[^(At)] = 0, which implies, in view of the estimate of lA, in 
()A.4jl . that the same holds if we replace 7 by lAc Thus dividing (jA.19jl by T and taking limits as 
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T —>■ oo, using the property y E^[V"a(-^T)] 0, and also using the hypothesis that v is optimal, 

we obtain 

/3* > [ (aVaix) + ^\Waix)+v{x)\'^') fi{dx). (A.21) 

jRd ^ ^ 

Since the subsequence {an} over which (|A.14p was obtained as a limit of (|A.3p was arbitrary, and 
given the estimate of VVq, in (IA.4p . we have lima^o aVa{x) = /3* for all x G Therefore, letting 
a 0 in (jA.2ip and applying again Fatou’s lemma, we obtain 

0 > / ^\W + v{x)\'^ 'q{dx). 

But since r) has a positive density, the equation above implies that v = —VF almost everywhere, 
thus completing the proof. □ 

Appendix B. Proofs of the results in Section 11.41 
We start with the proof of Lemma 11.161 

Proof of Lemma \1.16[ Suppose that M has a number q of eigenvalues on the open right half complex 
plane. Using a similarity transformation we can transform M to a matrix of the form diag(Mi, —M 2 ) 
where Mi G G are Hurwitz matrices. So without loss of generality, we 

assume M has this form. Let Si and S 2 be the unique symmetric positive definite matrices solving 
the Lyapunov equations S'lMi + MJSi = —I and S 2 M 2 + MJS 2 = —I, respectively. Extend these 
to symmetric matrices in by defining = diag(S'i,0) and ^2 = diag(0, S' 2 ), and also define, 
for a > 0, 

ifiix) ■= ip2(x) := ^ ^ . 

LetTi = diag(/(rf_g)x(d-g),Ogxq), and similarly r 2 = diag{0(^d-q)x{d-q)Jqxq)- Then, with £„/(x) := 
^Af{x) + (Mx + v{x),Vf{x)), we obtain 

Cv{l — (p 2 {x)) = aif 2 {x) (^trace{S 2 ) — 2a(x, S 2 X) + 1 X 2 x 1 "^ + 2{v{x), S 2 x)'^ 

> a(f2{x)(tTcace{S2) + \T2x\‘^ - ^\T2x\‘^ - 2a\\S2\\‘^\T2x\‘^ - 2||,S2||^|ii(a;)P) 

= a(/92(a;)(^trace(S'2) + (i - 2 q;||52|P)|T2xP - 2||52|P|u(x)P^ . (B.l) 

For the inequality in (jB.ll) we use 

2(v{x),S2x) = 2{S2v{x),T2x) > -\V2S2v{x)\‘^ 

> -i|r2x|2-2||S2fKx)|2. 

Using the analogous inequality for CnP 2 ix) and combining the equations we obtain 

Cy(p{x) > ae“"^*’'^i^^(^-trace(5i) + (5 + 2a\\Si\\'^)\Tix\‘^ - 2||5i||^|riu(x)p^ 

+ ae“"^^’'^2U ^trace(52) + (^ - 2a||,S2||^) |r2xl^ - 2\\S2\\‘^\T2v{x)\'^'^ 

> a(- trace(5i) + (i - 2a||5|p) jxp - 2||5||2|i;(x)l2) , (B.2) 

with S := diag(S'i, S' 2 ). 
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Using Ito’s formula on (IB.21) . dividing by a, and also using the fact that tp > 0 and Hv^Hoo = 2, 


we obtain 

E.. 


^^(-trace(5i) + - 2a\\Sf)\Xt\‘^ - 2\\Sf\viXt)\^) dt 


2 

< - 
a 


Dividing by T, letting T oo and rearranging terms, we conclude that |^|2 -g i^tegrable 

with respect to invariant probability measure under the control v for any a < 4 p|p 5 ^iid the 
following bound holds 

f u idx) < __^__ f \v(x)\‘^ a (dx) 

L ' ' ' - a- 2a||S|P) + (i - 2a||S|P) ' 

Taking limits as a 0, using monotone convergence, we obtain 

f |xp/i^(dx) < trace(5i) + 4||5||^ f \v{x)\‘^ py{dx). 

Js.d JR4 

The proof is complete. □ 


Proof Theorem \1.18[ It is well known that there exists at most one symmetric matrix Q satisfying 
(ll.26p - (ll.27l) |12l Theorem 3, p. 150]. For k > 0, consider the ergodic control problem of minimizing 


JK.iv) 


:= limsup 

T—^oo 


1 

T 


E 


f 

Jo 


IX 


(2 , i 

I ^ 2 


^i(x)nds 


(B.3) 


over V G ilssM, subject to the linear controlled diffusion 

Xt = Xo+ [ {MXs + v{Xs))ds + Wt, t > 0. (B.4) 

Jo 

As is also well known, an optimal stationary Markov control for this problem takes the form 
v{x) = —QkX, where Qk is the unique positive definite symmetric solution to the matrix Riccati 
equation 

Ql- M^Q^-Q^M = 2 kI. (B.5) 

Moreover, has the following property: Consider a deterministic linear control system x{t) = 
Mx{t) + u{t), with x,u £ M'^, and initial condition x(0) = xq. Let U denote the space of controls u 
satisfying /Q^|u(t)P dt < oo for all T > 0, and 4>f{xo) denote the solution of the differential equation 
under a control u £lA. Then 


(xo,QkXo) = min [ + 2K\(pf {xo)\'^) dt. 

ueU Jq \ / 


(B.6) 


For these assertions, see [121 Theorem 1, p. 147]. 

On the other hand, Qkx) is a solution of the associated HJB equation 

+ min (Mx + u, V4 'k(x)) + + n\x\^ = ^trace((5re). 


(B.7) 


The HJB equation (IB.7P characterizes the optimal cost, i.e., 

inf Jk(u) = dtrace(QK)- 
2^SilsSM 

Recall Definition 11.171 Since the stationary probability distribution of (|B.4I) under the control 
v{x) = —QkX is Gaussian, it follows by (IB.31) that G = Q^. minimizes 

Jg\k{M) '■= K trace(S g) + d trace(G Sc G"'') 



























CONTROLLED EQUILIBRIUM SELECTION 


41 


over all matrices G € G{M), where Eg is as in (jl.24p (note that Jc-oiM) = Jg{M) which is the 
right hand side of (jl.25l) l. Combining this with (IB.71) we have 

(B.8) 


inf Jg-,k.{M) = = itrace((5K). 

Geg(M) \ ^ 


By Lemma ll.161 we have 

trace(EQj < Co{l + Jq^-k{M)) 

= Co{l + \ trace(QK)) • (B.9) 

It also follows by (IB.OP that ~ Qk, is nonnegative definite if k! > k. Therefore Q^, has a unique 
limit Q as K \ 0. It is evident that Q is nonnegative semidehnite, and (IB.51) shows that it satisfies 
(jl.26l) . Since trace(EQ„) is bounded by (IB.9p . it follows that converges along some subsequence 
Atn 0 to a symmetric positive semidehnite matrix S. Thus (11.271) holds. However, (11.271) implies 
that S is invertible, and therefore, it is positive dehnite. In turn, (I1.27P implies that M — Q is 
Hurwitz. 

Since vg{x) = —Gx, G G Q{M), is in general suboptimal for the criterion Jk{v), applying 
Lemma 11.161 once more, we obtain 

Jq.(M) < Jq^-AM) < kCo{1 + Jg{M))+Jg{M) yG^Q{M). 

Therefore, we have 

MM) < JqMM) < nCo{l + MM))+MM), 
and taking limits as k \ 0, this implies by (jB.Sp that J^{M) = ^ trace((5). 

It remains to show that A^{M) = ^ trace((5). Let T be a unitary matrix such that Q := TQM 
takes the form Q = diag(0, ( 52 )) with Q G a positive dehnite matrix. Write the corresponding 
block structure of TMM as 


M := TMM = 


Mil Mi2 

M21 M22. 


with M 22 G Since M'^Q -\-QM = Q^, we obtain M'^Q + QM = Q^, and block multiplication 

shows that Q 2 M 21 = 0, which implies that M 21 = 0. Since M — Q is similar to M — Q the latter 
must be Hurwitz, which implies that Mu is Hurwitz. By block multiplication we have 

iV4<32 + Q 2 M 22 = Qi . (B.io) 

Since Q 2 is positive dehnite, the matrix —M 22 is Hurwitz by the Lyapunov theorem. Thus A^{M) = 
trace(M 22 ). Therefore, since Q 2 is invertible, we obtain by (IB.lOp that 

trace((5) = trace((52) = trace(M^ + M 22 ) 

= 2trace(M22) = 2A^{M). 

This proves part (a). 

Now let V G ilssM be any control. Let V{x) = ^^{x^Qx). Then V satishes (|B.7p with «: = 0, and 
we have 


^AI/(a:) + (Mx + h(a;), VH(x)) + = ^tTace{Q) + ^\Qx + v{x)\‘^ . 


(B.ll) 


Applying Ito’s formula to (|B.lip . and using the fact that fj,y has hnite second moments as shown 
in Lemma ll.161 and V is quadratic, a standard argument gives 


^'v{x)\^ — \\Qx + v{x)\^] yLy{dx) = itrace(Q). 


Thus ^\x{x)\'^ fj.y{dx) > ^trace((5) = Hence (11.281) holds. 


(B.12) 
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Suppose V is optimal, i.e., attains the infimum in (ll.28p . By (IB.12h . we obtain 

lim / \Qx + v{x)\'^ = 0. 

''NO JR-' 

Therefore, since jiy has a positive density, it holds that v{x) = —Qx a.e. in This completes the 
proof of part (b). 

We have shown that V{x) = ^{x, Qx) satisfies (I1.29P with ^ = 71+(M) and the associated process 
is positive recurrent. Therefore, as in the proof of Theorem II .41 for a bounded m, part (c) follows by 
Theorems 2.1-2.2 and Corollary 2.3 in [20]. Note that Hypothesis (A4) in |20| is easily satisfied for 
the linear problem. Since M is exponentially dichotomous, then as seen in the proof of Theorem l2.2l 
there exists symmetric matrices S and S, with S positive definite such that S + SM = S. 
Consider the function ipoix) := a{x,Sx), with a := 3 (||>9~^||||«S'||) Then 

:= ^ A(/5o(a:) + (Mx, V(/?o(x)) - i |V(/?o(a;)p 
= a trace 5 + a (x, 5x) — 2 a^ |5xp 
> I (2trace S'— (x, 5x)) . 

Thus A[^pa]{x) —)• 00 as |x| —>■ 00 . This completes the proof. □ 
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